The write was atomic. The event was not.

Business Challenge

The previous post established idempotency — the property that makes an at-least-once pipeline safe to consume. This one is about getting the events into that pipeline at all, which turns out to be the harder half.

An order is saved. The downstream system never hears about it. The handler is four lines, it is obviously correct, and it is the single most common correctness bug in event-driven systems:

save_order(order)            # write 1 — the database
publish("OrderPlaced", order) # write 2 — the broker

Two writes to two systems with no transaction spanning them. If the process dies between them, the order exists and the event does not. Swap the order and you get the opposite failure — an event for an order that was never saved — which is worse, because it is unrecoverable rather than merely late.

1There is no ordering of the two that is safe

Teams usually discover this by trying both orders and finding each one broken, then reaching for a retry. A retry does not help: the failure is that the process no longer exists to perform it. Wrapping them in application-level compensation does not help either, because the compensation is itself a write that can fail.

This is the dual-write problem, and it is not solvable at the call site. It is only solvable by making the two facts — the state change, and the intent to publish — land in one place that can commit or not commit as a unit.

Fix

Stop trying to coordinate two systems. Write the event into the same store, in the same transaction.

2The atomic write does not produce atomic events

This is the part that catches people who did everything right. You use TransactWriteItems, the entity and the outbox record commit together, and you assume the consumer sees them together. AWS says otherwise, and says it without hedging:

“This propagation occurs gradually: stream records from the same transaction might appear at different times and could be interleaved with records from other transactions. Stream consumers shouldn't assume transaction atomicity or ordering guarantees.”

The transaction is a guarantee about the write. It is not a guarantee about the stream derived from that write. A consumer that observes the outbox record before the entity it refers to is seeing supported behaviour.

Fix

Make each event self-contained, or make the consumer tolerate an entity that is not visible yet.

3An idempotent write emits nothing

The previous post argued for idempotent handlers. Combine that with a stream-based outbox and there is a trap sitting exactly at the join: “If you perform a PutItem or UpdateItem operation that does not change any data in an item, DynamoDB Streams does not write a stream record for that operation.”

A retry that correctly writes the same values a second time is a no-op, and a no-op is silent. If the first attempt wrote the row but died before the stream consumer ran, and the retry is a true no-op, there is no second chance to emit. The idempotency that protects the database is the thing that suppresses the event.

Fix

Give the outbox record a unique key per event, so writing it is always a genuine change.

Architecture

The pattern has three moving parts, and the interesting design work is at the joints rather than in any one of them.

Diagram: the dual-write problem and how an outbox built on DynamoDB addresses it, including where the guarantee stops. On the left, the broken shape: an application performs two writes, one to a database and one to a message broker, with no transaction spanning them, so a crash between the two leaves the state change without its event or the event without its state change. In the middle, the fix: a single TransactWriteItems call writes the entity and an outbox record together as one all-or-nothing operation of up to 100 actions, limited to one AWS account and one Region with an aggregate size of four megabytes. On the right, the relay: DynamoDB Streams derives change records from that same committed write, each stream record appears exactly once, and ordering is guaranteed per individual item rather than across a partition, with records retained for 24 hours before becoming subject to trimming. A panel across the bottom records where atomicity stops: AWS states that stream records from the same transaction might appear at different times and could be interleaved with records from other transactions, and that stream consumers should not assume transaction atomicity or ordering guarantees, so the consumer must treat each event as independently meaningful. A second note records that a write which does not change any data produces no stream record at all, so an idempotent no-op retry emits nothing. — The transaction removes the dual write. It does not extend atomicity into the stream — that boundary is where the consumer's design starts.

The transaction is real, and it is bounded

TransactWriteItems is “a synchronous and idempotent write operation that groups up to 100 write actions in a single all-or-nothing operation”, and the actions “are completed atomically so that either all of them succeed or none of them succeeds.” For an outbox that is exactly the primitive required: the entity and its event record commit together or not at all.

The bounds are worth knowing before you design around it. Up to 100 distinct items, in one or more tables, within the same AWS account and the same Region, with an aggregate size no greater than 4 MB. You cannot target the same item twice in one transaction. And the ACID guarantee is Regional only: “transactions aren't supported across Regions in global tables”, so a global table replica can show a partially propagated transaction.

It also costs double, by design. “DynamoDB performs two underlying reads or writes of every item in the transaction: one to prepare the transaction and one to commit the transaction.” An outbox therefore costs four write units per business event, not one — two items, doubled — and that arithmetic belongs in the design note rather than in a bill.

A failed transaction is not a free transaction

AWS is explicit that the capacity “is consumed even when a transaction does not succeed” — a cancelled call still consumes the underlying write capacity for the items it attempted. Worth pairing with the retry behaviour: a rejected item-level request fails the whole call with TransactionCanceledException, and “if that request fails, AWS SDKs do not retry the request.” Contention against a hot item is therefore paid for twice over, in capacity and in a retry you have to write yourself.

The stream is the relay, and it has its own contract

The reason this pattern fits DynamoDB so well is that the relay does not have to be built. The stream is derived from the committed write, so there is no second system to keep in step, and it carries a guarantee most transports do not: “Each stream record appears exactly once in the stream.”

Read that precisely. It is a statement about the stream's contents, not about delivery to your consumer — a Lambda event source mapping can still hand you the same record twice, which is why the previous post exists. Ordering is similarly specific: guaranteed “at the level of an individual item… not across an entire partition.” Two different orders have no defined relative order. The same order's three state changes do.

Why This Architecture Holds Up

Self-contained events are what survive the boundary

Once you accept that a consumer may see the outbox record before the entity, and may see two transactions interleaved, the design consequence is forced: the event has to carry what the consumer needs rather than a pointer to fetch it.

An event that says “order 4471 changed, go and read it” is a lookup against a table that may not show the change yet. An event carrying the order's relevant state is independently meaningful. This is the same reasoning that makes the NEW_AND_OLD_IMAGES view type the useful default for an outbox: the record is complete without a round trip.

The 24-hour floor is the whole recovery window

Stream data has “a 24-hour lifetime”, and records older than that are “susceptible to trimming (removal) at any moment.” That is not a tuning knob — there is no setting to extend it. If a consumer is broken over a long weekend, the events it never processed are gone, and the only recovery is to re-derive state from the table itself. An outbox on DynamoDB Streams therefore needs a documented reconciliation path, not just a DLQ, because the DLQ only holds what was successfully delivered and failed — not what expired before delivery was attempted.

Consumer parallelism is capped by the shard, not by your concurrency setting

Throughput planning here surprises people who are used to scaling consumers freely. AWS advises that “no more than two processes at most should be reading from the same stream's shard at the same time”, since more “can result in throttling.”

With Lambda the shape is cleaner but still bounded: one function instance per open shard by default, raisable via ParallelizationFactor “up to 10” while “still preserving the order of changes for each item.” That last clause is the one that makes the setting safe to use — ordering is preserved per item, which is the only ordering the stream promised in the first place.

Shard lineage is the other constraint: an application “must always process a parent shard before it processes a child shard.” Lambda handles this for you. A hand-rolled consumer that ignores it will process records out of order during exactly the traffic spike that caused the shard to split.

Where the outbox is genuinely the wrong answer

AWS's own transaction guidance argues against reaching for this reflexively: “Don't group operations together in a transaction if it's not necessary… Simpler transactions improve throughput and are more likely to succeed.”

If the only consumer needs the change and nothing else, a plain write with a stream consumer is already sufficient — the stream is the outbox, derived from the same committed write, with no second record and no doubled capacity. The explicit outbox record earns its cost when the event's shape must differ from the entity's: a published contract that should not change when the storage model does, an event that aggregates several items, or one that must carry intent the row does not record.

Key Architecture Decisions

Decision	Choice	Reasoning
Dual write	Never; one transaction	No ordering of two writes to two systems is crash-safe, and retries cannot help a dead process.
Explicit outbox record	Only when the event shape differs	Otherwise the stream is already the outbox. AWS advises against unnecessary transactions.
Outbox record key	Unique per event	A write that changes nothing produces no stream record, so a reused key emits silence on retry.
Event payload	Self-contained, not a pointer	Consumers may see the event before the entity is readable. AWS says not to assume otherwise.
Stream view type	`NEW_AND_OLD_IMAGES`	Makes the record complete without a round trip, and cannot be changed without recreating the stream.
Consumer semantics	Idempotent, order-tolerant across items	Exactly-once applies to the stream's contents, not to delivery. Ordering holds per item only.
Recovery beyond 24 hours	Reconcile from the table	Stream retention is fixed at 24 hours with no extension. Expired records were never delivered.
Capacity planning	Four write units per event	Two items, and DynamoDB writes each twice — prepare and commit — including on failure.

Two details that decide whether the retry story works

The transaction's own idempotency window is ten minutes. A client token “is valid for 10 minutes after the request that uses it finishes. After 10 minutes, any request that uses the same client token is treated as a new request.” Reusing it with changed parameters inside the window returns IdempotentParameterMismatch. That is a tighter bound than the hour the Powertools guard defaults to and far tighter than a next-morning replay — the same window-versus-incident mismatch the previous post ended on, one layer down.

Disabling a stream is not reversible in the way people assume. Re-enabling creates a new stream with a different descriptor, the view type cannot be edited in place, and the old stream's data remains readable for 24 hours and then goes. Changing what an outbox publishes is therefore a migration with a cutover, not a configuration change.

Closing Thought

The dual-write problem is usually presented as a puzzle with a clever solution, and the outbox pattern is presented as that solution. On AWS the mechanics are almost disappointingly easy — a transaction, a stream, a Lambda — which is exactly why the failures land somewhere else.

They land at the boundary where one guarantee stops and the next has not started. The transaction guarantees the write. The stream guarantees its own contents, once each, ordered per item. Neither guarantees that a consumer sees a coherent transaction, and AWS says so in a sentence most people never read: stream consumers shouldn't assume transaction atomicity or ordering guarantees.

The pattern that survives is the one that stops asking the transport for consistency it never offered. Make each event true on its own. Make the consumer idempotent, because delivery is at-least-once. Assume ordering only where it was promised, which is per item. Then the atomic write is doing the one job it can actually do — making sure the event exists at all — and everything after it is designed for the world as the documentation describes it.

Next in this series

Application patterns — retries, backoff, jitter and the thundering herd: why every client retrying politely still produces a synchronised stampede, what the AWS SDKs do by default, and how a retry budget differs from a retry policy.

AWS Architecture Series #53 — The write was atomic. The event was not.