The retry you did not write

Business Challenge

The last four posts were about money. This one opens a different block — the patterns that decide whether a distributed system is correct — and it starts with the property that almost every other one depends on.

A customer is charged twice for one order. The investigation is uncomfortable because everything looks right. Nobody wrote retry logic. The queue is FIFO, chosen specifically to avoid this. The handler reads cleanly, and replaying the event by hand produces one charge, not two.

Three assumptions failed, and none of them is a coding error. They are all misreadings of what the platform promised.

1The retry was issued on your behalf

You do not have to write a retry to get one. For asynchronous invocation, “by default, Lambda retries a failed asynchronous invocation up to two times.” One event, up to three executions, no retry code anywhere in the repository.

SQS does the same thing from the other direction. If a consumer does not delete a message before the visibility timeout expires — “due to application errors, crashes, or connectivity problems” — the message “becomes visible again in the queue and can be retrieved by the same or a different consumer for another processing attempt.” A handler that charges a card and then crashes before deleting has already done the work, and the platform will hand the same message to somebody else.

AWS states the obligation this creates about as plainly as documentation ever does: “ensure that your function's code can handle the same event multiple times without causing duplicate transactions or other unwanted side effects.”

Fix

Assume every handler runs more than once. The question is not whether, it is what that costs.

2FIFO deduplicates the send, not the processing

This is the assumption that does the most damage, because the page is titled “Exactly-once processing” and the feature really does work — at the thing it does. “FIFO queues help you avoid sending duplicates to a queue.” If a producer retries SendMessage within the 5-minute deduplication interval, no duplicate enters the queue.

That is producer-side. It says nothing about a message being delivered to a consumer twice, and AWS is explicit that choosing FIFO does not buy you that: in both standard and FIFO queues, “due to the at-least-once delivery model of Amazon SQS, there's no absolute guarantee that a message won't be delivered more than once during the visibility timeout period.”

Fix

Treat FIFO as ordering plus send-side deduplication. Consumer-side idempotency is still yours to build.

3The guard had already expired

Say the team did everything right and added an idempotency key with the Powertools utility. It “allows you to retry operations within a time window with the same input, producing the same output” — and that window has a default. “By default, we expire idempotency records after an hour (3600 seconds).”

An hour is generous for a retry storm and far too short for an incident. The duplicate charge did not come from a burst; it came from a dead-letter queue replayed the next morning, long after every record had expired. To the handler, the replay was new work, because that is exactly what an expired key means.

Fix

Set the expiry from how long a replay could realistically be delayed, not from the default.

Architecture

There are three places a duplicate can be suppressed, they cover different things, and the useful discipline is knowing which one is doing the work.

Diagram: the three layers at which a duplicate request can be suppressed in AWS, and the boundary each one has. At the top, three sources issue retries that the application did not write: Lambda retries a failed asynchronous invocation up to two times by default, Amazon SQS makes a message visible again for another processing attempt if it is not deleted before the visibility timeout expires, and the client itself may retry after a timeout. Below that, three suppression layers are shown with their limits. The transport layer is SQS FIFO deduplication, which prevents duplicates being sent into the queue within a five-minute deduplication interval, but is producer-side only and does not guarantee single delivery to a consumer. The API layer is the Amazon EC2 client token, a string of up to 64 ASCII characters whose idempotency is scoped either regionally or zonally, so the same token used in another Region or Availability Zone launches additional instances, and which returns an IdempotentParameterMismatch error if retried with different parameters. The application layer is an idempotency key stored in DynamoDB, where a conditional write is idempotent when the condition checks the same attribute being updated, and where the Powertools utility expires its records after one hour by default and raises IdempotencyAlreadyInProgressError when a second invocation arrives with the same payload before the first has completed. A panel at the bottom records the gap: a dead-letter queue replayed after a multi-hour incident falls outside both the five-minute deduplication interval and the one-hour record expiry, so the replay is indistinguishable from new work. — Three layers, three different boundaries. The transport layer is send-side, the API layer is scoped by Region or Availability Zone, and the application layer expires.

The API layer has a scope, and the scope is not obvious

AWS's own definition is the clearest statement of the property anywhere in the documentation: “Idempotency ensures that an API request completes no more than one time. With an idempotent request, if the original request completes successfully, any subsequent retries complete successfully without performing any further actions.” EC2 implements it with a client token, a unique string of up to 64 ASCII characters.

The part that surprises people is that the token is not globally unique in effect. EC2 idempotency is Regional or zonal, and the same client token used in a different Region — or, for a zonal request, a different Availability Zone — launches more instances. A retry helper that fails over to a second Region while reusing the token does not deduplicate. It doubles.

There is a useful guardrail alongside it: reusing a token with different parameters fails with IdempotentParameterMismatch rather than silently doing something else. That is the right failure — an error beats an ambiguous success — and it is worth knowing that AWS also treats some actions as idempotent by default with no token at all, including TerminateInstances and AssociateAddress.

The application layer is a conditional write, and the condition matters

Underneath any idempotency library is the same primitive: a write that succeeds only if this work has not already been recorded. DynamoDB states the rule precisely — conditional writes “can be idempotent if the conditional check is on the same attribute that is being updated.” The qualifier is the whole sentence. A condition on a different attribute than the one you are changing is not idempotent, and it will look like it works until two retries interleave.

The counter-example AWS supplies itself

An atomic counter is the canonical thing that cannot be retried: “With an atomic counter, the updates are not idempotent. In other words, the numeric value increments or decrements each time you call UpdateItem.” AWS is direct about where that is acceptable — website visitor counts, where slight overcounting does not matter — and where it is not: “An atomic counter would not be appropriate where overcounting or undercounting can't be tolerated (for example, in a banking application).” If a balance is maintained by an atomic counter, retry safety was lost at the schema, not in the handler.

Why This Architecture Holds Up

Concurrency is a separate problem from repetition

Most idempotency discussions assume the duplicate arrives after the first attempt finished. The harder case is two in flight at once, and a key-based guard alone does not resolve it: both requests look up the key, both find nothing, both proceed.

This is why the Powertools utility raises IdempotencyAlreadyInProgressError “if you receive multiple invocations with the same payload while the first invocation hasn't completed yet”, and its reasoning is worth adopting verbatim: “Since we don't know the result from the first invocation yet, we can't safely allow another concurrent execution.”

A guard that only records completions is a half-guard. The record has to be written when work starts, not when it succeeds, which means the store holds three states — in progress, completed, expired — and the in-progress state has to fail closed.

The windows are the design, not the defaults

Line the boundaries up and the gap is obvious. SQS FIFO deduplicates a resent message within 5 minutes. Powertools expires an idempotency record after 1 hour. A visibility timeout defaults to 30 seconds and can be extended, but only to “a maximum limit of 12 hours from when the message is first received”, and extending it “doesn't reset this 12-hour limit.”

Now put an incident against that scale. A bad deployment runs for two hours. Failed messages land in a DLQ. Someone triages in the morning and replays the queue — the standard, correct operational response. Every one of those messages is outside the deduplication interval and outside the record expiry, so every replayed message is new work by definition.

The replay is the case the guard was bought for

This is the uncomfortable part. Idempotency is usually justified by pointing at retries and redeliveries — events measured in seconds. But the scenario that actually produces duplicate charges is the deliberate human replay of a DLQ hours later, and that is the one case where every default window has already closed. If the expiry is not set from the maximum plausible replay delay, the guard protects against the cheap failure and not the expensive one.

The key has to be the business event, not the message

One more trap sits in the transport layer. SQS content-based deduplication uses “a SHA-256 hash to generate the message deduplication ID using the body of the message — but not the attributes of the message.” Two sends with an identical body and different attributes are the same message to the deduplication logic.

The same reasoning applies one layer up, in the opposite direction. If your application-level key is derived from the whole payload, a retry carrying a fresh timestamp or a new trace ID hashes differently and sails straight through. The key must be the identity of the business event — the order ID, the payment reference — not a digest of the envelope that happened to carry it.

Key Architecture Decisions

Decision	Choice	Reasoning
Baseline assumption	Every handler runs more than once	Lambda retries async invocations twice by default and SQS redelivers on visibility-timeout expiry. Neither needs your code.
FIFO queue	For ordering, not for exactly-once	FIFO avoids sending duplicates. AWS states there is no absolute guarantee against redelivery in either queue type.
Idempotency key	The business event identifier	A payload digest changes when a timestamp or trace ID does, and the retry passes straight through.
Record expiry	Maximum plausible replay delay	The one-hour default covers retry storms, not a DLQ replayed the next morning — which is the expensive case.
Concurrent duplicates	Write the record at start, fail closed	A guard that records only completions lets two in-flight requests both proceed.
Persistence primitive	Conditional write on the attribute being updated	DynamoDB's own qualifier. A condition on a different attribute is not idempotent.
Counters	Not for anything that must reconcile	AWS states atomic counter updates are not idempotent and names banking as the case where that is unacceptable.
Cross-Region retries	Do not reuse an EC2 client token	EC2 idempotency is Regional or zonal. The same token in another Region launches more instances.

Two costs worth putting in the design note

The guard is not free, including when it works. A rejected duplicate still writes: “If a ConditionExpression evaluates to false during a conditional write, DynamoDB still consumes write capacity from the table.” A retry storm that the guard correctly suppresses is a storm of consumed capacity. That is a far better outcome than duplicate charges, and it should still be sized rather than discovered.

Idempotency changes what a timeout means to the caller. Once an operation is safely repeatable, a client that times out can simply retry, which is the entire point. Before that property exists, a timeout is genuinely ambiguous and the only safe client behaviour is to stop and ask a human. Idempotency is what converts an ambiguous failure into an ordinary one, and that is why it belongs near the bottom of the dependency list for everything else in this block.

Closing Thought

Idempotency gets taught as a coding technique — check a key, skip if seen. The mechanics are genuinely that simple, which is why the interesting failures are never in the mechanics. They are in the boundaries: a deduplication interval measured in minutes, a record expiry measured in an hour, a client token scoped to one Region, a condition attached to the wrong attribute.

Every one of those boundaries is a default that somebody accepted rather than chose. The default is usually right for the failure everyone imagines, which is a fast retry, and wrong for the failure that actually reaches a customer, which is a human replaying a queue after an incident has been resolved.

The architectural move is to stop asking whether the system is idempotent and start asking for how long, and within what scope. Those two answers are the design. The key check is just where it gets written down.

Next in this series

Application patterns — the outbox pattern and the dual-write problem: why writing to a database and publishing an event is two writes that cannot be made atomic, what DynamoDB Streams and transactional outboxes actually guarantee, and how the idempotency property built here becomes the thing that makes an at-least-once event pipeline safe to consume.

AWS Architecture Series #52 — The retry you did not write