Business Challenge
Six posts of application patterns have assumed a step can fail and be retried. This one is about what happens when it cannot — when the work already succeeded, the next step failed, and the only way back is forward.
An order reserves stock, charges a card, books a courier slot. Three services, three datastores, no transaction spanning them. The courier booking fails. Stock is reserved and the card is charged, and nothing will roll that back because there was never anything to roll back to.
The saga is the accepted answer: each step gets a compensating action, and on failure you walk backwards running them. The pattern is sound. Three things about implementing it on AWS are not obvious, and the first one is the reason compensations fail in production.
A catcher's ResultPath “determines what input the catcher
sends to the state specified in the Next field.” Leave it
out and Step Functions is explicit about what happens: “if you don't specify the
ResultPath field, it defaults to $, which
selects and overwrites the entire input.”
So the compensation state is reached exactly as designed, and arrives holding an error object
with a Cause string. The order id, the payment id, the reservation id
— everything it needs to undo anything — has been replaced by a description of what
went wrong.
It fails in the most expensive way available: the workflow looks correct, the transition happens, and the undo runs against nothing.
FixSet ResultPath on every catcher, always. "$.error" adds the error and keeps the input.
States.ALL does not mean all
The wildcard reads like total coverage and is not. “The
States.ALL error type must appear alone in a
Catcher and cannot catch the
States.DataLimitExceeded terminal error or
Runtime error types.”
And on the one that bites hardest: “A
States.Runtime error isn't retriable, and will always cause the
execution to fail. A retry or catch on States.ALL won't catch
States.Runtime errors.”
Those come from things like applying a path to a null payload — ordinary data-shape bugs.
A saga whose entire safety net is one States.ALL catcher has no net
under the failures most likely to occur while it is being built.
Name the errors you expect explicitly, and treat States.ALL as the last of several, not the only one.
Catchers are per-state. AWS says so directly: they are “available for Task, Parallel and Map states, but not for top-level state machine execution failures.”
A saga that times out as a whole, or hits an uncatchable runtime error, does not run its compensations. It stops, holding reserved stock and a charged card, and nothing inside the workflow will do anything about it.
AWS names the ways out: let the caller handle it, nest the work in a child workflow so the parent
can catch it, or “listen for TIMED_OUT events from Standard
workflows with an EventBridge bus and invoke an action to handle the failed execution.”
Put the saga in a child workflow, or wire TIMED_OUT to something that compensates. Do not leave it unhandled.
Architecture
The useful frame is that Step Functions gives you two different mechanisms and they answer two
different questions: Retry asks will this work if I try again?,
and Catch asks what do I do now that it will not?
Retry first, then catch — and the defaults are generous
The ordering is fixed and worth knowing: “When a state has both
Retry and Catch fields, Step Functions uses
any appropriate retriers first. If the retry policy fails to resolve the error, Step Functions applies
the matching catcher transition.”
The retrier defaults are IntervalSeconds of 1,
MaxAttempts of 3, and a
BackoffRate that “increases by 2.0”.
AWS's own worked example makes the shape concrete: with an interval of 3 and a backoff of 2, the
retries land at three seconds, six seconds, and 12 seconds.
Two details matter for a saga specifically. JitterStrategy defaults to
NONE — the previous post on retries was about what
that costs a fleet, and it is off here unless you set FULL. And
“retries are treated as state transitions”, which on a Standard workflow is the
billing unit. A generous retry policy on a step that is reliably broken is a bill as well as a delay.
This is the one line to take away. A catcher with "ResultPath": "$.error" “adds the error output to the input” and passes the whole thing onward; a catcher without it replaces the input entirely. The first is a compensation that knows which order to undo. The second is a compensation that knows only that something failed. Nothing in the console warns you which one you built, and both look identical on the workflow graph.
The workflow type decides whether a compensation runs once
Compensations are side effects, usually irreversible ones, so how many times they run is not an academic question. The two workflow types answer it differently, and “the workflow type can not be updated after you create a state machine” — so this is decided once, at creation.
Standard follows “an exactly-once model, where your tasks and states are
never run more than once, unless you have specified Retry behavior”,
which AWS says suits “orchestrating non-idempotent actions, such as
starting an Amazon EMR cluster or processing payments.” That is a saga's description of
itself.
Express uses “an at-least-once model, so an execution could potentially run more than once”, suited to “idempotent actions”. Run a saga on asynchronous Express and the refund may be issued twice — which is the same at-least-once contract post #52 was about, one layer up.
Why This Architecture Holds Up
The state has to live somewhere, and only one type keeps it
A saga is a sequence with memory: to compensate step three you must know steps one and two ran. Step Functions states the difference plainly — for Standard, “execution state internally persists between state transitions”; for Express, “execution state doesn't persist between state transitions.”
That is the architectural reason a saga belongs on Standard, ahead of duration or cost. The same asymmetry shows up in naming: Standard “automatically returns an idempotent response on starting an execution with the same name as a currently-running workflow”, while for asynchronous Express “idempotency is not automatically managed. Starting multiple workflows with the same name results in concurrent executions.” Two concurrent sagas for one order is a compensation running against a step the other saga is still performing.
Real sagas wait. A fraud review needs a human; a fulfilment step needs a job to finish. Both are service integration patterns, and AWS is explicit that “Express Workflows do not support Job-run (.sync) or Callback (.waitForTaskToken) service integration patterns” — nor Distributed Map, nor Activities. Combined with the five-minute ceiling, a saga with any human in it cannot be Express. And since the type is immutable, discovering that later means building a new state machine rather than changing a setting.
Compensation is not rollback, and some steps cannot be compensated
Everything above is mechanical. The part that is not mechanical is that a compensating action is a new forward action that approximates an undo, and the approximation is not always available.
Releasing a stock reservation is a genuine inverse — the system returns to its prior state. Refunding a payment is not: the charge and the refund both exist, the customer sees both, and a statement now shows activity that the original transaction model would never have produced. Sending the confirmation email has no inverse at all. The compensation for a sent email is a second email apologising for the first, which is not a rollback; it is customer service implemented in a state machine.
This is why step ordering is a design decision rather than an implementation detail. Put the steps with real inverses first and the irreversible ones last, so that the further a saga gets, the less likely it is to need an undo it cannot perform. A saga that emails the customer before charging the card has ordered itself so that the most common failure produces the least reversible mess.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Catcher input | ResultPath on every catcher |
The default is $, which overwrites the whole input with the error output. |
| Error matching | Named errors, then States.ALL last |
The wildcard catches neither States.Runtime nor States.DataLimitExceeded. |
| Workflow-level failure | Child workflow, or TIMED_OUT on EventBridge |
Catchers do not exist for top-level execution failures. Otherwise nothing compensates. |
| Workflow type | Standard | Exactly-once, state persists between transitions, and it is the type AWS names for non-idempotent actions. |
| Compensation handlers | Idempotent anyway | A redrive resets retry counts, and a compensation may be reached more than once. |
| Retry policy | Explicit, with JitterStrategy: FULL |
Jitter defaults to NONE, and retries bill as state transitions. |
| Step ordering | Reversible first, irreversible last | The later a saga fails, the more likely its compensations are approximations rather than inverses. |
| Human-in-the-loop steps | Standard, decided up front | .waitForTaskToken is unsupported on Express, and the type is immutable. |
Two error names worth knowing precisely
States.TaskFailed is nearly a wildcard: it “acts as a wildcard
that matches any known error name except for
States.Timeout.” A catcher built on it silently excludes the
timeout case, which for a saga is among the most likely failures — a downstream service that
hangs rather than erroring.
And States.Timeout changes identity across a boundary: if a nested state
machine throws it, the parent receives States.TaskFailed instead. A
parent catching only States.Timeout to handle a child's timeout will not
match it.
Closing Thought
Two-phase commit is not coming back, and the saga is what replaces it — not as an equivalent, but as an admission. A transaction offers atomicity: nothing partially happened. A saga offers something weaker and honest: everything happened, and then some of it was undone as well as it could be.
Step Functions implements the orchestration well, and the orchestration was never the hard part. The hard parts are a default that hands your compensation an error where the order used to be, a wildcard that is not a wildcard, a workflow-level failure with no handler inside the workflow, and steps whose compensation is a second email rather than a reversal.
None of those are visible on the workflow graph. The graph of a saga with
ResultPath set and a saga without it are the same picture, and only one of
them can undo anything. That is worth remembering the next time a state machine diagram is offered as
evidence that a design is sound.
Application patterns — CQRS, and the commoner case where it is overkill: what separating reads from writes actually buys, why the read model's staleness is the whole design rather than a defect in it, and the far more frequent situation where one model and a well-chosen index would have been the answer.
Comments