The stop condition halts the experiment, not the damage

Business Challenge

#108 ended on the gap AWS names inside its own passing status: an estimate is not a timed recovery, so go and test it. FIS is the service that closes that gap, and it opens with a warning rather than a feature list:

“AWS FIS carries out real actions on real AWS resources in your system.”

Which makes the guardrail the interesting part. An experiment template contains “the actions, targets, and stop conditions for the experiment”, and the stop condition is the only one of the three whose job is to limit harm. It is worth knowing exactly what it bounds before running anything in production.

1The guardrail is optional, and its absence is a documented value

A stop condition is not built in. It is something you supply: “A stop condition is a mechanism to stop an experiment if it reaches a threshold that you define as an Amazon CloudWatch alarm.” And you can decline. The documentation gives the shape explicitly:

"stopConditions": [ { "source": "none" } ]

“The following example indicates that the experiment template does not specify a stop condition.” That is a first-class, documented configuration — not an omission the API rejects. An experiment template with no guardrail is a valid template.

This matters because of what an experiment report looks like afterwards. A completed experiment with no stop condition triggered reads exactly like a completed experiment that had no stop condition to trigger. The distinction lives in the template, not in the result.

Fix

Treat "source": "none" as a finding in template review, not a default. Then record, next to every experiment result, which alarms were armed — because that list is what turns the experiment completed into evidence. Without it the result is a statement that FIS finished running, which is true of a template targeting nothing in particular.

2Completed without an alarm is two findings, and five alarms cover twenty actions

AWS is clear about where a stop condition comes from: “To create a stop condition, first define the steady state for your application or service.” The alarm encodes your definition of steady state, so the experiment's verdict inherits whatever that definition covers. An experiment that terminated instances and tripped nothing tells you your application survived — or that the alarm watched CPU while the failure was in a queue depth nobody graphed.

The budget is fixed and small. “Stop conditions per experiment template | Each supported Region: 5” against “Actions per experiment template | Each supported Region: 20” — four actions per guardrail if you use the whole template, and neither number is adjustable. In fact no FIS quota in the table is adjustable, which is unusual enough to plan around rather than discover.

Fix

Write the steady-state definition down before the template, and make it the thing under review rather than the action list. A useful test: name the signal that would move first if this experiment went badly, and check an armed alarm watches it. If the honest answer is that nobody knows which signal moves first, that is the experiment's actual finding and it is available before running anything.

3Stopping is not reverting, and you cannot resume

“If a stop condition is triggered during an experiment, AWS FIS stops the experiment. You cannot resume a stopped experiment.”

Both halves matter. Stopping ends the injection; it does not reverse what the injection already did. An action that terminated instances has terminated them, and the stop condition's achievement is that it terminated fewer. Recovery is whatever your Auto Scaling group, your Kubernetes controller or your runbook does next — which is the thing the experiment was testing, now running under a half-finished fault.

And because a stopped experiment cannot be resumed, a tripped guardrail costs you the run. The experiment ends in a state that is neither the steady state nor the full fault, which is the hardest state to draw a conclusion from.

Fix

Plan the recovery path as a deliverable of the experiment, not an afterthought — and pair the run with the SOP that #108's Policy met definition asks you to test and time, because this is where that timing actually gets measured. Then treat a stop-condition trigger as a result to investigate rather than a run to repeat: it is the one outcome that tells you an alarm you wrote fired on a fault you chose.

Architecture

Diagram: An AWS FIS experiment template contains actions, targets and stop conditions, and FIS carries out real actions on real AWS resources. The stop condition is the only one of the three whose job is to limit harm, and it is a CloudWatch alarm you define rather than something built in. Declining to have one is a documented first-class value, written as stopConditions with source set to none, so an experiment template with no guardrail is a valid template and a completed experiment with no alarm triggered reads identically to one that had no alarm to trigger. When a stop condition does fire, FIS stops the experiment and you cannot resume it, but stopping ends the injection rather than reversing what it already did. Targets are resolved once: FIS identifies all targets at the start of the experiment before starting any actions, and uses the resources it selected for the entire experiment, failing if none are found. Targets can be identified by resource ARNs, by tags where multiple tags are combined with AND and an empty tag value is not a wildcard, by filters where multiple filters are combined with AND and multiple values within one filter with OR, or by resource parameters. Three selection modes scope the result: ALL runs on all targets, COUNT of n runs on that many chosen at random, and PERCENT of n runs on that percentage chosen at random, rounding down, so five instances at fifty percent gives two and four instances at five percent selects nothing, and a scope identifying no resources fails the experiment. If multiple targets use the same resource type, FIS can select the same resource more than once. Every FIS quota is non-adjustable: five stop conditions and twenty actions per template, which is four actions per guardrail, ten parallel actions, five active experiments, five hundred templates, twelve hours maximum for an experiment or an action which is 720 minutes of billable action time, and 120-day retention of completed experiment data. Per-action target caps include 300 instances for terminate, 600 for reboot and 1,000 pods for pod delete. Billing is per minute that an action runs, from start to finish, based on the number of target accounts.

Targets are resolved once, at the start, and frozen

“AWS FIS identifies all targets at the start of the experiment, before starting any of the actions in the actions set. AWS FIS uses the target resources that it selects for the entire experiment. If no targets are found, the experiment fails.”

This is the right design and worth contrasting with #106. A resource group is a query answered whenever something reads it, so the set an automation acts on can change under it. An FIS experiment resolves its targets once and holds them, which means the blast radius is fixed for the duration even if somebody edits a tag mid-run. For a service that performs destructive actions, freezing the set is the correct trade.

What it does not fix is the set being whatever the query returned at that instant. If targets came from tags, the experiment's scope is the tag state at start time, and the AND semantics are the same ones #106 described: “If you specify more than one tag, all specified tags have to be present on the target resource for it to be selected (AND).” A fourth tag filter narrows the experiment. So does a typo.

Three selection modes, and two ways to select nothing

Mode	What it does	Edge
`ALL`	“Run the action on all targets”	Blast radius is whatever the query matched
`COUNT(n)`	“the specified number of targets, chosen from the identified targets at random”	Random, so two runs are not the same experiment
`PERCENT(n)`	“the specified percentage of targets, chosen from the identified targets at random”	Rounds down, and can round to nothing

The rounding is documented with examples worth reading literally. “If you have an odd number of resources and specify 50%, AWS FIS rounds down. For example, if you add five Amazon EC2 instances as targets and scope to 50%, AWS FIS rounds down to two instances.” And the floor: “if you add four Amazon EC2 instances and scope to 5%, AWS FIS can't select an instance.”

That second case does not quietly run on zero resources, which would be the dangerous outcome: “Regardless of which selection mode you use, if the scope that you specify identifies no resources, the experiment fails.” A failed experiment is a visible result. A green experiment that touched nothing would not be.

The same resource can be targeted twice

“If you define multiple targets using the same target resource type, AWS FIS can select the same resource multiple times.”

Two target definitions over the same resource type are two independent random draws from overlapping populations. A template with COUNT(1) for a CPU stress action and COUNT(1) for a network latency action can put both faults on one instance, which is a different experiment from the one most people have in mind when they write two targets. Worth checking in any template that uses more than one target of the same type.

The quotas, all of them fixed

Quota	Default	Adjustable
Stop conditions per experiment template	5	No
Actions per experiment template	20	No
Parallel actions per experiment	10	No
Active experiments	5	No
Experiment templates	500	No
Experiment duration in hours	12	No
Completed experiment data retention in days	120	No

The retention one shapes a programme more than it looks. 120 days of completed experiment data, not adjustable, means a quarterly resilience review can see the previous quarter and not the one before it. If you report on experiment history, export it.

The per-action target caps are also fixed and vary by how destructive the action is, which reads as deliberate: “Target Instances for aws:ec2:terminate-instances | Each supported Region: 300” against “Target Instances for aws:ec2:reboot-instances | Each supported Region: 600”, and “Target Pods for aws:eks:pod-delete | Each supported Region: 1,000”. You may reboot twice as many instances as you may terminate.

What it costs, and the unit that surprises people

“You are charged per minute that an action runs, from start to finish, based on the number of target accounts for your experiment.”

Per action-minute, multiplied by target accounts — not per experiment and not per resource. A twelve-hour experiment is 720 minutes of billable action time per account, and ten parallel actions bill as ten. A long soak across a large organisation is the shape that costs real money, and it is the shape a team reaches for when testing slow degradation.

Why This Architecture Holds Up

Opening with a warning instead of a benefit

The first callout on the service's front page is that FIS acts on real resources and that you should run experiments in pre-production first. A chaos engineering service that led with its dashboard would be easier to adopt and worse. Putting the risk in front of the feature is the correct ordering for a tool whose entire function is to break things.

Freezing the target set is the right call for destructive actions

Resolving targets once and holding them means an experiment cannot grow mid-run because a deployment added instances or someone edited a tag. The alternative — re-evaluating continuously, as a resource group does — would make the blast radius unbounded in exactly the situation where you least want that.

Failing on an empty target set rather than passing

An experiment that matched no resources fails. It would have been easy to let it complete with nothing done, and that would have produced the worst artefact in this series: a green result that proves nothing, indistinguishable from one that proves something. FIS declines to produce it.

Key Architecture Decisions

Decision	Choose this	Because
Template with no stop condition	Treat `"source": "none"` as a review finding	It is a valid documented configuration, so nothing rejects it for you
Recording an experiment result	Record which alarms were armed alongside it	Completed-without-alarm means nothing until you know what the alarm watched
Allocating the five stop conditions	Arm the signals that move first, not the ones that are easiest to graph	Five is fixed against twenty actions, and not adjustable
Expecting a trigger to protect you	Plan the recovery path as part of the experiment	FIS stops the experiment; it does not reverse completed actions, and you cannot resume
Multiple targets of one resource type	Check whether they can overlap	FIS can select the same resource more than once across target definitions
Using `PERCENT` on a small fleet	Compute what it rounds to before running	It rounds down, and below one resource the experiment fails rather than runs
Reporting on experiment history	Export it on a schedule	120-day retention, not adjustable
Long soak experiments	Price them per action-minute per target account first	Twelve hours is 720 billable action-minutes, multiplied by parallel actions and accounts

The audit worth running

List your experiment templates and grep for "source": "none". Every hit is an experiment that runs without a guardrail. Some of those will be deliberate and fine — a read-only action in a sandbox account — but the list should be short and each entry should have a reason somebody can state.

For the rest, open the alarms and ask what they measure. An alarm on instance CPU is not a steady-state definition for a latency-sensitive API. This is the check that decides whether your experiment results are evidence or just completion records.

Then look for templates with two or more targets of the same resource type. Work out whether the populations overlap and whether you intended the same resource to be able to take both faults. If you did, say so in the template description; if you did not, the fix is a filter.

Closing Thought

This run of posts has kept finding instruments whose output is narrower than it reads. #108 was the sharpest case because the narrowing was a single word — estimated. FIS is the opposite shape: it states its limits plainly and in the right places, and the risk is not that it overclaims but that a reader under-reads.

The thing worth carrying is the asymmetry between the two halves of a guardrail. The half that stops an experiment is automatic, fast and reliable. The half that decides when to stop is a CloudWatch alarm somebody wrote on a Tuesday, against a definition of steady state that may never have been reviewed. The service supplies the mechanism. You supply the judgement, and the judgement is the part that determines whether a green experiment report means anything.

So the practical ending is the same shape as #108's: record what was watching, next to what happened. An experiment result without its armed alarms is a log line. With them, it is a measurement — which is the thing #108 said you still owed.

Next in this series

Reliability — Amazon Application Recovery Controller: what a readiness check actually checks, why zonal autoshift moves traffic on AWS's signal rather than yours, and what a routing control is when the thing it routes around is the control plane you would use to operate it.

AWS Architecture Series #109 — The stop condition halts the experiment, not the damage