Home› Blog› AWS Architecture Series #54 — The herd is spread, not smaller…
AWS Architecture AWS Architecture Series

AWS Architecture Series #54 — The herd is spread, not smaller

A dependency returns 503 for ninety seconds. Every client retries correctly — exponential backoff, full jitter, sensible attempt limits, none of it hand-written — and the dependency still cannot recover, because politeness is a property of each client alone and nothing in the SDK coordinates across a fleet.

Sep 16, 2026 11 min read
Verified against current vendor documentation on 16 September 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

The last two posts made a pipeline safe to consume: idempotent handlers, then events that exist at all. This one is about the traffic those retries generate, which is a different problem and a more counter-intuitive one.

A dependency starts returning 503 and keeps it up for ninety seconds. Every client behaves impeccably. Exponential backoff, full jitter, a sane attempt limit — and not one line of it hand-written, because the SDK does it. The dependency still struggles to recover, and the graph shows request volume rising while the service is down.

Three things are true at once here, and only the first is widely understood.

1Jitter spreads the burst; it does not shrink it

The mechanism is exactly as advertised, and AWS explains the problem it solves in the documentation's own words: without the random multiplier, “all clients that hit an error at the same time would retry at the same time, creating a burst of retry traffic (the ‘thundering herd’ problem).”

Then read the worked example carefully. “Suppose 1,000 clients all receive a 503 at the same moment. Full jitter distributes their first retries uniformly across a 50 ms window instead of having all 1,000 retry at exactly 50 ms.”

A thousand requests still arrive. They arrive spread across fifty milliseconds rather than stacked on one instant, which genuinely helps a service absorb them — but the load is unchanged. For a transient error the base delay is 50 ms, and that is a narrow place in which to distribute a large fleet.

Fix

Expect jitter to smooth a spike, not to reduce volume. Reducing volume is a different control.

2The circuit breaker is per client, not per fleet

The retry quota is the better-kept secret and the more useful mechanism. It is a token bucket: 500 tokens, a transient retry costs 14, a throttling retry costs 5, and a request that succeeds first time restores 1. Drain it and the SDK stops retrying and returns errors immediately.

That is a real circuit breaker, and it is exactly the behaviour you want during an outage. But its scope is the sentence people skip: “The token budget is typically scoped to a single SDK client instance… It is not shared across processes or hosts.”

A fleet of a thousand hosts therefore holds a thousand independent budgets, each discovering the outage privately and each draining on its own schedule. Nothing aggregates them.

Fix

Treat it as per-client self-protection. Fleet-level backpressure has to come from somewhere else.

3You are probably not running any of this yet

This is the detail that reframes the whole page, and it sits in a notice at the top of it. The documented behaviour “requires opting in until it becomes the default behavior. Set AWS_NEW_RETRIES_2026=true in your environment.”

And the consequence of not doing so is not cosmetic: “Without this setting, your SDK uses pre-2026 retry behavior, which differs in backoff timing, retry quota costs, and service-specific defaults.”

Every number below — the 500-token budget, the 14-token cost, the 50 ms base, the 20-second cap — describes the opted-in system. Reading the page and changing nothing leaves you running something else.

Fix

Decide deliberately. Set the variable and adopt these semantics, or know precisely which ones you are not getting.

Architecture

There are two independent defences in the SDK and one decision that sits above both of them. They operate at different points in the request lifecycle, and neither reaches past the client it runs in.

Diagram: the two retry defences an AWS SDK provides and the boundary each one has. At the top, the backoff formula: the delay equals a uniformly random value between zero and one, multiplied by the smaller of twenty thousand milliseconds and the base delay doubled for each retry. The base delay is 50 milliseconds for transient errors and 1,000 milliseconds for throttling errors, and no individual delay exceeds the 20 second cap, although with the default of three max attempts the cap is never reached. In the middle, the first defence, full jitter, which spreads retries uniformly across the backoff window rather than stacking them on one instant, illustrated by AWS's own example of a thousand clients receiving a 503 simultaneously and having their first retries distributed across a 50 millisecond window. Beside it, the second defence, the retry quota token bucket, with a capacity of 500 tokens, a cost of 14 tokens for a transient retry and 5 for a throttling retry, restoring 1 token when a request succeeds on the first attempt, and beginning to drain above roughly 22 percent sustained transient failures or 32 percent throttling failures. A panel across the bottom records the shared limitation: the token budget is scoped to a single SDK client instance and is not shared across processes or hosts, so a fleet holds one independent budget per client and nothing aggregates them, meaning jitter smooths the spike and the quota protects the individual caller while neither reduces total fleet load. A final note records that all of this behaviour is opt-in until it becomes default, requiring the environment variable AWS underscore NEW underscore RETRIES underscore 2026 set to true.
Two defences, both local to one client. Jitter changes when a request arrives; the quota changes whether it is sent at all.

The formula, and what the default hides

The whole backoff scheme is one line: 

delay = random(0, 1) × min(20,000 ms, base_delay × 2^retry)

base_delay is 50 ms for transient errors and 1,000 ms for throttling, which is a deliberate distinction: a throttle means the service actively rejected you and needs time, while a connection reset usually resolves in milliseconds.

The 20-second cap is real but mostly theoretical in practice, and AWS says so plainly: “With the default of 3 max attempts (1 initial request + 2 retries), the backoff cap is never reached.” Three attempts is the default for most clients — DynamoDB and DynamoDB Streams use 4, with a 25 ms transient base rather than 50 ms.

The error taxonomy decides the delay, and one rule is easy to get wrong

Classification runs on error code first and HTTP status second, which produces a case worth knowing: “An HTTP 5XX with a throttling error code is classified as a throttling error, not a transient error, even though 5XX errors are normally transient.” The practical effect is a twentyfold difference in base delay for two responses that look alike in a status-code dashboard — so a latency investigation that groups by status will not explain why some 5XX retries wait a second and others wait fifty milliseconds.

The service can override your backoff, and it is trusted to

Some AWS services return an x-amz-retry-after header. When present, the SDK uses the server's delay, “clamped to a minimum of the computed backoff delay and a maximum of the computed backoff delay plus 5,000 ms”, an effective ceiling of 25 seconds.

One detail in that mechanism is a small lesson in protocol design: “The SDK does not apply jitter to this value, because the service is expected to jitter it.” Responsibility for spreading the herd moves to the only party that can see the whole herd. That is the right place for it, and it is precisely the capability an individual client does not have.

Why This Architecture Holds Up

The quota is a better circuit breaker than most people hand-build

The token economics are deliberately asymmetric. A transient retry costs 14 tokens; a throttling retry costs 5. AWS's reasoning is that transient errors “often indicate a service-wide problem” where “continued retrying is unlikely to succeed”, while a throttle only signals that the service “needs more time before the request can succeed.” Retrying into a broken service is priced higher than retrying into a busy one.

The thresholds that result are specific: the budget begins draining above roughly 22% sustained transient failures, or 32% for throttling. Below that, first-try successes refill it faster than retries drain it. And because “the budget's starting balance of 500 tokens provides a buffer”, a brief severe spike does not trip it at all — only a sustained one does. That is the behaviour you would want from a circuit breaker and rarely get from one written in an afternoon.

But both defences stop at the client boundary

Jitter spreads retries within one client's window. The quota stops one client retrying into a failure. Neither one knows another client exists, and the budget is explicitly “not shared across processes or hosts.” A thousand hosts each hold a full budget, each independently discover the outage, and each independently spend 500 tokens learning it. The SDK protects the caller from wasting its own resources. It was never designed to protect the callee from the aggregate, and expecting that of it is the mistake that makes a fleet-wide incident surprising.

Adaptive mode is the one control that throttles the first request

Standard mode has a hard boundary: “The retry quota never delays or blocks the initial request. Only retries are affected.” So no matter how badly a dependency is failing, standard mode sends every first attempt.

Adaptive mode is the exception — it “can delay or block the initial request, not just retries, when throttling is detected.” That sounds like the fleet-level answer, and it is not, for a reason worth stating: “The rate limiter operates per SDK client instance. All requests from a client share the same rate limit, regardless of which API operation or resource they target.”

So one throttled resource slows every call that client makes, including calls to healthy resources. AWS is direct that this is not a general default and recommends it only for a client targeting a single resource with frequent throttling — a batch processor against one table, not a shared service client. Use it where a client has one job.

Long polling gets an exception, and the reason generalises

There is a special case for SQS.ReceiveMessage and the Step Functions and SWF polling operations: when the quota is depleted and retries are blocked, “the SDK still applies a backoff delay before returning the error.”

The reasoning is worth borrowing. These are called in a tight loop, so returning an error instantly would mean the application “would then immediately send the next request, spiking client CPU usage and generating additional traffic.” Failing fast is correct for a request a human is waiting on, and actively harmful inside a polling loop. If you build your own fail-fast path, check which of those two shapes your caller has.

Key Architecture Decisions

Decision Choice Reasoning
Adopting the 2026 behaviour Set it deliberately, per environment It is opt-in. Without AWS_NEW_RETRIES_2026=true the SDK uses pre-2026 timing, quota costs and defaults.
Retry mode Standard AWS's own recommendation for all workloads, and the only one standardised across SDKs.
Adaptive mode Single-resource clients only The rate limiter is per client, so one throttled resource slows that client's calls to every other.
Legacy mode Migrate off it No standardised quota, and behaviour differs per language. Not even available in .NET, Go, Kotlin, Rust, Swift or JavaScript.
Max attempts Leave at 3 Raising it multiplies fleet load during an outage. The 20-second cap is never reached at the default anyway.
Hand-written retry loops Delete them They nest with the SDK's, multiplying attempts, and they carry no token budget.
Fleet-level protection Server-side, not client-side The budget is not shared across processes or hosts. Only the service sees the aggregate.
Polling loops Keep a delay on the failure path Fail-fast in a tight loop spikes CPU and traffic. AWS special-cases the long-polling operations for this reason.

The multiplication nobody plans for

The most common way to turn a recoverable incident into a long one is layered retries. An SDK client configured for 3 attempts, inside an application retry loop of 3, behind a queue redelivery, produces an attempt count that nobody wrote down and nobody intended — and only the innermost layer has a token budget. The outer layers retry at full rate regardless of how the service is doing.

The discipline is to retry at exactly one layer and let the others surface the error. If the SDK is doing it, the application should not be. If the application must, the SDK's attempts should be set to 1, which AWS documents as the way to disable retries entirely.

Closing Thought

Retry advice is usually a list of techniques — exponential backoff, add jitter, cap the attempts — and all of it is correct. The SDK implements every item on that list, to a standard most teams would not reach by hand, and it does it without being asked.

What the list omits is that each technique is a property of one client. Jitter decides when a client retries. The token budget decides whether it retries. Neither decides how many clients there are, and during an incident that is the only number that matters to the service on the other end.

So the architectural question is not whether your retries are polite. Assume they are; the defaults are good. The question is what happens when a thousand polite clients discover the same failure in the same second — and the answer has to come from somewhere with fleet-wide visibility: a server-side throttle, a queue that absorbs the arrival rate, a x-amz-retry-after the service chooses itself. The client library has done its job. It cannot do that one.

Next in this series

Application patterns — backpressure, and what a queue is really telling you: why queue depth is a lagging indicator of a problem that started upstream, the difference between a buffer and a shock absorber, and when adding consumers makes an overload worse rather than better.

Topics: AWS Architecture AWS Architecture Series

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent