Week 24 - Chaos Engineering with AWS FIS: The Safety Net That Could Never Close

Lab Overview

You fit a smoke alarm, press the test button, and it beeps. You now believe the kitchen is covered. But the button only tests the speaker — it says nothing about whether the sensor can smell smoke.

The problem. A chaos experiment's safety net is a CloudWatch alarm wired in as a stop condition. It shows a green OK, and everyone assumes it would catch a real failure. Almost nobody tests whether it can.

The objective. Deliberately break an ECS service with AWS Fault Injection Service and measure how much damage runs before the stop condition closes, comparing a fast alarm against a slow one.

The outcome. That measurement is impossible on this workload, for four separate reasons, each of which presented in the console as a successful run. Those four reasons are worth more than the number would have been.

Cloud Services and Concepts

Architecture and Workflow

1. An FIS experiment template names a target (tasks tagged FISTarget, 50% of them), an action (aws:ecs:stop-task) and a stop condition.
2. The action stops two of the four tasks. ECS notices and begins replacing them without being asked.
3. The ALB deregisters the stopped tasks from its target group.
4. HealthyHostCount dips from 4 to 2. UnHealthyHostCount stays at 0 — a deregistered target is absent, not unhealthy.
5. Two alarms read that metric: one evaluating a single 60-second window, one evaluating two 300-second windows.
6. Either alarm, if it fires, halts the experiment as a stop condition.
7. ECS finishes replacing the tasks and the service returns to 4 healthy.

Architectural decisions

Alarm on HealthyHostCount, not UnHealthyHostCount. The obvious metric cannot see this fault at all, for the reason in step 4.

Statistic Minimum, not Average. Over a 300-second window a one-minute dip to 2 averages to roughly 3.6 and never crosses a threshold of 4.

Both alarms come from one definition, differing only in period and datapoints_to_alarm. If anything else drifted, the comparison would be measuring something other than patience.

aws:ecs:stop-task rather than the aws:ecs:task-* family. The richer actions — CPU stress, network latency — all require SSM-managed tasks, which on Fargate means a sidecar in every task definition. That is a lot of machinery for a lab about stop conditions. This choice has a consequence, covered in Troubleshooting.

Prerequisites

One quota decides whether the design fits. A FIS experiment template allows 20 actions; the repeated-fault template here uses 13. Check before building:

aws service-quotas list-service-quotas --service-code fis --region us-east-1 \
  --query "Quotas[?contains(QuotaName,'per experiment template')].[QuotaName,Value]"

Network assumption. The service runs in the default VPC with public IPs, and the ALB shares those subnets — which keeps the lab to 19 resources and no NAT gateway. It also rules out aws:network:disrupt-connectivity as an alternative fault, because that action targets subnets and would cut the load balancer along with the tasks.

No prerequisite labs. Week 24 is self-contained. The service module is adapted from Week 21's blue/green ECS build, but nothing from that week needs to exist.

Implementation

Nineteen resources: an ECS cluster, service and task definition; an ALB, listener and target group; two security groups; two alarms; three FIS experiment templates; four IAM roles and policies; two log groups. The lab guide carries the full Terraform, the exact commands and the expected output. This is the shape of it.

week-24-chaos-engineering-fis/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the two modules
|   +-- modules/service/    ECS Fargate service, ALB, and the two alarms
|   +-- modules/chaos/      FIS role, log group, three experiment templates
|
+-- scripts/                starts an experiment and records what happened
|
+-- docs/                   findings, figure sources, the commands behind them

The service. Four Fargate tasks behind an ALB, tagged so FIS can find them. One line carries more weight than it looks:

propagate_tags = "SERVICE"

FIS selects tasks by tag, and a task does not inherit its service's tags unless told to — without it the experiment targets nothing and still reports success.

The two alarms (service module) differ in exactly two lines:

period              = 60     # fast      |  300   # typical
datapoints_to_alarm = 1      #           |    2

Three experiment templates (chaos module): the same fault guarded by nothing, by the fast alarm, and by the slow alarm.

Measurement. A recording script samples the experiment state, both target counts and both alarm states every two seconds, and keeps sampling after FIS reports finished.

Validation and Evidence

Four claims are worth separating, because three of them passed while the fourth — the one the lab exists to test — could not even be measured.

Deployment succeeding is not the lab working. Before injecting anything, the service has to be genuinely serving traffic.

Evidence 1 — the metric the alarms read never moved

Procedure: stop 50% of tasks, then query both host-count metrics over the same window.
Expected: UnHealthyHostCount rises while tasks are down.
Observed: it reported 0.0 for every one-minute datapoint, across a window in which the service demonstrably halved.

Evidence 2 — six successful actions, one effect

Procedure: deliver the fault in seven rounds two minutes apart, giving it a duration a stop condition could interrupt.
Expected: each round removes half the running tasks.
Observed: all six rounds that ran reported "Action was completed". Only the first stopped anything.

Evidence 3 — the outage was shorter than the metric's sampling interval

Observed: the recorder saw 2 healthy targets from t=28s to t=45s, a real 17-second loss of half the service. CloudWatch's one-minute Minimum over the same window did not show it:

ECS replaced both tasks in about 40 seconds. The dip never reached the metric — and it is not deterministic: an earlier, identical run did record a 2.0. This proves the service recovered faster than its own observability could sample it.

What that means for the original question. It is unanswerable here. The fault heals in 17 seconds, the metric samples every 60, and the action completes in 373 milliseconds. There is no window in which any stop condition could have acted.

Security and Cost

Security

No static credentials. HCP Terraform assumes an AWS role by OIDC; no access key exists in the workspace.

Least privilege with confused-deputy conditions. The FIS role can describe one cluster, stop tasks in that cluster only, read the two stop-condition alarms, and write to its own log group. Its trust policy pins the source account and ARN.

One permission could not be narrowed. logs:PutResourcePolicy is account-level and cannot be scoped to a log-group ARN, and FIS experiment logging needs it. It sits on *. Worth stating rather than hiding.

Blast radius is declared before the fault is: one tagged service, one cluster, a fixed percentage. That is the control that makes this safe to run — and as this lab showed, it should be verified by reading Resources targeted, not by trusting a green experiment state.

Not production-ready as built. Tasks run in public subnets with public IPs to keep the lab small; in production they belong in private subnets behind NAT or VPC endpoints with the ALB public. And a stop condition is not one of this build's working controls — it is the thing the lab disproved. If you rely on one, test that the alarm beneath it can physically reach ALARM for the fault you inject.

Cost

$1.26 measured, not estimated — unblended cost for the single UTC day the lab existed.

Almost all of those action-minutes were aws:fis:wait — padding added so the fault would last long enough for a stop condition to interrupt. The stop-task actions that did the damage ran 373 ms each and rounded to nothing. The fault was free; the waiting was the bill. FIS charges elapsed action time, not damage done.

The remaining 19 cents is Fargate compute, the load balancer and public IPv4 addresses for about two hours. A FIS experiment report would have been another $5 — four times the rest of the lab — so the evidence here is the experiment detail page and the recorded CSVs instead.

Troubleshooting and Lessons

An alarm on UnHealthyHostCount can never fire for a stopped ECS task

Problem. Both stop conditions sat at OK through a run that halved the service.
Root cause. FIS stops the task and ECS deregisters it from the target group. UnHealthyHostCount counts targets that are registered and failing a health check; a deregistered target is absent, not unhealthy.
Resolution. Alarm on HealthyHostCount below desired count, statistic Minimum.
Lesson. A green alarm proves the alarm exists, not that it can fire.

A stop condition needs an action with a duration

Problem. The guarded experiments finished before either alarm could evaluate.
Root cause. aws:ecs:stop-task ran for 373 milliseconds. The fastest possible alarm here needs 60 seconds.
Resolution. Deliver the fault in rounds separated by aws:fis:wait, which has a one-minute floor and needs no IAM permissions.
Lesson. "Add a stop condition" is not by itself a safety measure. Check the action lasts long enough to be interrupted.

FIS resolves its targets once

Problem. Six repeated kill rounds, six "Action was completed", one actual effect.
Root cause. FIS resolved the target set at t=0 and held it. Later rounds re-issued StopTask against already-stopped tasks, which succeeds. Not a tagging failure — the replacement tasks carried FISTarget correctly.
Resolution. Expect only the first round of a repeated fault to act against a self-healing target.
Lesson. Read Resources targeted. A completed action is not an effective one.

The recorder stopped watching before the damage landed

Problem. The first run recorded a flat line of 4 healthy targets through a run that demonstrably halved the service.
Root cause. FIS reports completed when its last action returns. It reported completed at 22:56:31; ECS started replacements at 22:57:08 and the originals reached STOPPED at 22:57:41, seventy seconds later.
Resolution. Keep sampling through an "aftermath" phase until the service is whole again.
Lesson. A tool's terminal state describes the tool, not the system.

An HCP run can apply code you did not push

Problem. A run reported "no changes" while the broken IAM policy it was meant to fix was still live.
Root cause. A run created through the API without a configuration version reuses the last version HCP ingested.
Resolution. Pin the configuration version whose commit SHA matches HEAD, and refuse to run if it never arrives.
Lesson. Only reading the live policy in AWS caught this.

Cleanup and Key Takeaways

One destroy run removed all 19 resources. But a run reporting success is a claim about what it did, which is the kind of claim this lab spent four findings learning not to trust. So the check queries AWS for every resource type the lab created:

Nothing here is shared, so there is nothing to preserve. Left running, the ALB, the four Fargate tasks and the public IPv4 addresses all keep charging by the hour. The FIS templates cost nothing, but a template aimed at a cluster that no longer exists is worth deleting anyway.

Key takeaways

• A green alarm proves it exists, not that it can fire. Both stop conditions here were structurally unable to detect the fault they guarded, and looked identical to working ones.
• A stopped ECS task is deregistered, not unhealthy. Alarm on HealthyHostCount below desired, statistic Minimum.
• FIS resolves targets once. A repeated action does not re-select, so every round after the first can be a no-op reporting "Action was completed".
• A workload can out-heal its observability. A real 17-second outage never reached a metric sampled once a minute — recorded on one run, missed on an identical one.
• FIS bills elapsed action time, not damage. 81% of this lab's cost was waiting for an alarm that was never going to notice.

References

• The full lab guide — prerequisites, Terraform, commands, validation, cleanup
• FINDINGS.md — the four findings with complete evidence
• AWS FIS actions reference — which actions have a duration, and which need an SSM agent
• Stop conditions for AWS FIS
• AWS FIS pricing
• Terraform aws_fis_experiment_template