Home› Blog› Week 21 - Blue/Green on ECS: The Rollback That Nev…
AWS Weekly Lab AWS Terraform

Blue/green on ECS, and the rollback that never had to happen

Release a new version with no downtime, then deliberately ship a broken one to find out which part of the machinery actually protects users. It was not the part built for the job.

Verified against current vendor documentation on 3 October 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.
AWS Platform Engineering Lab · Week 21 of 52

Lab Overview

You have a website running and you want to put a new version live. The obvious way is to stop the old one and start the new one — which means a gap where the site is down, and no way back except doing the whole thing again in reverse.

The problem. Blue/green is the other way: start the new version alongside the old, wait until it looks healthy, then point the front door at it. The half everyone builds is the traffic switch. The half almost nobody tests is the pointing back.

The objective. Run a containerised service behind a load balancer, release a new version, then deliberately ship a broken one — to find out which part of the machinery actually protects users.

The outcome. Users were protected. Not by the rollback mechanism built for the job, which never fired at all.

DifficultyIntermediate
Lab runtimeRan about 3 hours before teardown
Measured cost$0.1552, 73% of it the load balancer
DeploymentTerraform via HCP Terraform with OIDC
Full lab guideGitHub — prerequisites, Terraform, commands, validation, cleanup

Cloud Services and Concepts

ServicePurpose hereWhat it is
Amazon ECS on FargateRuns the containers and, since 2025, performs the blue/green deployment itself — no CodeDeploy needed.Runs containers without managing servers. You declare how many copies (tasks) you want and ECS keeps that many running.
Application Load BalancerThe front door. One listener rule points at whichever target group is live.Spreads traffic across the tasks. Its target group is the register of which tasks exist and which are passing health checks.
Target groups (blue and green)Two interchangeable slots. ECS rewrites the listener rule to switch between them.A target group is the list of backends a load balancer may send to. Two of them let the same listener point at one version or the other.
CloudWatch alarmThe configured rollback trigger, watching 4xx responses.Reads a metric, summarises each fixed window with a statistic, and fires after N bad windows. Not a live tripwire.

Blue and green are slots, not fixed roles. ECS put the first deployment in green, which is worth knowing because Terraform names blue as the primary. Nothing is permanently "the old one".

A health check and an alarm protect against different things. The health check asks "can this task serve a request at all" before any user reaches it. The alarm asks "are real users getting errors" after they do. Which one catches your fault determines what you actually learned.

Architecture and Workflow

aws AWS Cloud Region: us-east-1 Client curl / browser 1 Application Load Balancer one listener rule 2 blue target group the version serving traffic green target group the version being tested 3 ECS service Fargate tasks CodeDeploy moves the rule 4 CloudWatch alarms during bake 5 A rollback is the same move in reverse: the rule points back at blue, and the old tasks were never stopped.
One listener rule decides which version is live. Blue keeps running, so going back is the same move in reverse.
  1. A client reaches the Application Load Balancer, which has one listener rule.
  2. That rule points at the blue target group, holding the current version.
  3. A deployment starts a new task set in the green target group alongside it.
  4. The new tasks must pass their health check before the load balancer will send them anything.
  5. Once healthy, ECS rewrites the listener rule to point at green. That rewrite is the switch.
  6. A CloudWatch alarm on 4xx is configured to trigger automatic rollback if the new version starts failing for real users.

Design decisions

ECS native blue/green, not CodeDeploy. ECS does this itself now, which takes one service out of the failure path.

Rollback keyed on 4xx, not 5xx. The deliberate fault returns 404s, so the alarm had to be able to see them. It still never fired — for the reason in Troubleshooting.

Public subnets with public IPs, to avoid NAT costs. A known gap: production puts the tasks in private subnets.

Prerequisites

RequirementDetail
AWS accountPermission for ECS, ELB, IAM, CloudWatch and VPC resources
Terraform>= 1.10, with an AWS provider supporting ECS native blue/green
HCP TerraformWorkspace with OIDC to AWS. No static keys
Regionus-east-1

Budget awareness before you start: unlike most labs in this series, this one has a real hourly meter. The load balancer bills whether or not anyone visits — roughly $16/month if left standing. Plan to tear it down the same day.

No prerequisite labs. Week 21 is self-contained.

Implementation

An ECS cluster and service on Fargate, an ALB with one listener and two target groups, the IAM roles, and a CloudWatch alarm. The lab guide carries the full Terraform and the exact deployment commands.

week-21-bluegreen-ecs/
|
+-- terraform/
|   +-- environments/dev/   HCP backend, OIDC, wires the modules
|   +-- modules/network/    VPC, subnets, security groups
|   +-- modules/service/    ECS cluster and service, ALB, both target groups
|   +-- modules/deploy/     the rollback alarm and deployment configuration
|
+-- scripts/                deploy, break, and watch from the client side
Only the directories that matter. The full repository is on GitHub.

One line decides whether a failed deployment gives up:

deployment_circuit_breaker { enable = true, rollback = true }
# off by default

Without it, a failing deployment does not stop — it retries, cycling tasks indefinitely.

HCP Terraform run list for week-21-dev showing the applied run and an earlier errored run
Twenty resources. The errored run above it is real — see Challenges.
The green target group in the AWS console with one registered task, showing 1 healthy and 0 unhealthy on port 8080
One of the two slots, holding the live version. Its twin sits empty until a deployment fills it. That pair is the whole of blue/green — everything else is ordinary container plumbing.

Validation and Evidence

A deployment that completes is not a deployment that was safe. The test that matters is the broken release.

TestExpectedActualStatus
Healthy release switches trafficListener rule moves to the new groupMoved; no downtime observed from the clientPass
Client sees no gap during switchContinuous 200sContinuous 200sPass
Broken release is kept from usersNo user traffic reaches itHealth check failed: Target.ResponseCodeMismatch [404]; no traffic sentPass
Alarm triggers automatic rollbackAlarm fires, deployment revertsAlarm never left OK; no state change in its historyFail — and that is the finding
Failed deployment gives up on its ownRevertsRetried for over ten minutes until reverted by handFail — circuit breaker was off
The ECS console during a deployment showing the new task set in progress alongside the active one
Mid-deployment: both versions running, traffic still on the old one.
Terminal output from the polling script for both deployments: 62 requests served v1 and 86 served v2 with no failures, then 214 requests all served by the old version during the broken deployment
The client's view of both releases. No failed request in either.
The ECS console showing the new deployment failing, with the target reported unhealthy due to a response code mismatch
Target.ResponseCodeMismatch — "Health checks failed with these codes: [404]".

Security and Cost

Security

No static AWS credentials. HCP Terraform assumes its role via OIDC.

Tasks reachable only through the ALB. The task security group accepts traffic from the load balancer's security group and nothing else.

Scoped IAM. Two AWS managed policies, one per role, no inline wildcards.

Known gap: tasks run in public subnets with public IPs to avoid NAT costs. Production puts them in private subnets.

Cost

Billed $0.1552 for about three hours, read from Cost Explorer once AWS posted the charges: Elastic Load Balancing $0.1126 and ECS/Fargate $0.0425 across 29–30 September, with $0.00 on the 1st — which is what a verified teardown looks like on a bill.

The load balancer was 73% of it, for a service that served a few hundred requests. That is the shape worth remembering: the meter is the front door, not the containers behind it. About $0.05/hour with both versions running, roughly $16/month for an ALB left up.

Troubleshooting and Lessons

The rollback never happened, and that is the finding

Problem. The deliberately broken version was deployed and the automatic rollback alarm never fired.
Root cause. The broken version failed its health check (Target.ResponseCodeMismatch — 404), so ECS never sent it any traffic. No client request reached it, so there were no errors, so the alarm stayed OK and rolled back nothing. Alarm history confirms no state change during the deployment.
Resolution. None needed — users were protected. But by the health check, not by the mechanism built for the job.
Lesson. If you only ever test rollback with a fault your health check catches, you have tested the health check. Use a fault that passes the health check and fails for users.

A failing deployment does not give up on its own

Problem. The failed deployment cycled tasks for over ten minutes.
Root cause. deploymentCircuitBreaker is off by default. The deployment retried rather than reverting.
Resolution. Enable the circuit breaker with rollback.
Lesson. Two separate safety features, both off or inert by default, neither doing what the name suggests without configuration.

A successful apply is not a successful deployment

Problem. terraform apply returned applied with zero new tasks running and the old version still serving.
Root cause. Terraform hands off to ECS and returns; the deployment continues asynchronously.
Resolution. Watch the deployment from the client side, not the apply output.
Lesson. Gate a pipeline on that apply and you have gated on nothing.

Cleanup and Key Takeaways

Destroyed via HCP, then verified independently of the teardown script: no load balancer, no cluster, no target groups, no leftover network interfaces, workspace at zero resources.

bash scripts/cleanup.sh
# CLEAN - nothing from Week 21 remains.

Everything here bills by the hour, so an incomplete teardown is not a tidy loose end — it is roughly $16/month. The $0.00 on the following day is the evidence that the destroy was complete.

Key takeaways

  • ECS does blue/green itself now. No CodeDeploy for new work — one less service in the failure path.
  • Test rollback with a fault your health check cannot see. Otherwise you are testing the health check and learning nothing about the alarm.
  • Turn on the deployment circuit breaker. Without it a failing deployment retries instead of reverting.
  • The load balancer is the cost. It bills hourly with no traffic at all, so tear it down.
  • Watch a deployment from the client's side. The console shows task sets; only a caller shows the service.

References

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent