📋 In This Post
Lab Overview
You have a website running and you want to put a new version live. The obvious way is to stop the old one and start the new one — which means a gap where the site is down, and no way back except doing the whole thing again in reverse.
The problem. Blue/green is the other way: start the new version alongside the old, wait until it looks healthy, then point the front door at it. The half everyone builds is the traffic switch. The half almost nobody tests is the pointing back.
The objective. Run a containerised service behind a load balancer, release a new version, then deliberately ship a broken one — to find out which part of the machinery actually protects users.
The outcome. Users were protected. Not by the rollback mechanism built for the job, which never fired at all.
| Difficulty | Intermediate |
| Lab runtime | Ran about 3 hours before teardown |
| Measured cost | $0.1552, 73% of it the load balancer |
| Deployment | Terraform via HCP Terraform with OIDC |
| Full lab guide | GitHub — prerequisites, Terraform, commands, validation, cleanup |
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| Amazon ECS on Fargate | Runs the containers and, since 2025, performs the blue/green deployment itself — no CodeDeploy needed. | Runs containers without managing servers. You declare how many copies (tasks) you want and ECS keeps that many running. |
| Application Load Balancer | The front door. One listener rule points at whichever target group is live. | Spreads traffic across the tasks. Its target group is the register of which tasks exist and which are passing health checks. |
| Target groups (blue and green) | Two interchangeable slots. ECS rewrites the listener rule to switch between them. | A target group is the list of backends a load balancer may send to. Two of them let the same listener point at one version or the other. |
| CloudWatch alarm | The configured rollback trigger, watching 4xx responses. | Reads a metric, summarises each fixed window with a statistic, and fires after N bad windows. Not a live tripwire. |
Blue and green are slots, not fixed roles. ECS put the first deployment in green, which is worth knowing because Terraform names blue as the primary. Nothing is permanently "the old one".
A health check and an alarm protect against different things. The health check asks "can this task serve a request at all" before any user reaches it. The alarm asks "are real users getting errors" after they do. Which one catches your fault determines what you actually learned.
Architecture and Workflow
- A client reaches the Application Load Balancer, which has one listener rule.
- That rule points at the blue target group, holding the current version.
- A deployment starts a new task set in the green target group alongside it.
- The new tasks must pass their health check before the load balancer will send them anything.
- Once healthy, ECS rewrites the listener rule to point at green. That rewrite is the switch.
- A CloudWatch alarm on 4xx is configured to trigger automatic rollback if the new version starts failing for real users.
Design decisions
ECS native blue/green, not CodeDeploy. ECS does this itself now, which takes one service out of the failure path.
Rollback keyed on 4xx, not 5xx. The deliberate fault returns 404s, so the alarm had to be able to see them. It still never fired — for the reason in Troubleshooting.
Public subnets with public IPs, to avoid NAT costs. A known gap: production puts the tasks in private subnets.
Prerequisites
| Requirement | Detail |
|---|---|
| AWS account | Permission for ECS, ELB, IAM, CloudWatch and VPC resources |
| Terraform | >= 1.10, with an AWS provider supporting ECS native blue/green |
| HCP Terraform | Workspace with OIDC to AWS. No static keys |
| Region | us-east-1 |
Budget awareness before you start: unlike most labs in this series, this one has a real hourly meter. The load balancer bills whether or not anyone visits — roughly $16/month if left standing. Plan to tear it down the same day.
No prerequisite labs. Week 21 is self-contained.
Implementation
An ECS cluster and service on Fargate, an ALB with one listener and two target groups, the IAM roles, and a CloudWatch alarm. The lab guide carries the full Terraform and the exact deployment commands.
week-21-bluegreen-ecs/
|
+-- terraform/
| +-- environments/dev/ HCP backend, OIDC, wires the modules
| +-- modules/network/ VPC, subnets, security groups
| +-- modules/service/ ECS cluster and service, ALB, both target groups
| +-- modules/deploy/ the rollback alarm and deployment configuration
|
+-- scripts/ deploy, break, and watch from the client side
One line decides whether a failed deployment gives up:
deployment_circuit_breaker { enable = true, rollback = true }
# off by default
Without it, a failing deployment does not stop — it retries, cycling tasks indefinitely.
Validation and Evidence
A deployment that completes is not a deployment that was safe. The test that matters is the broken release.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Healthy release switches traffic | Listener rule moves to the new group | Moved; no downtime observed from the client | Pass |
| Client sees no gap during switch | Continuous 200s | Continuous 200s | Pass |
| Broken release is kept from users | No user traffic reaches it | Health check failed: Target.ResponseCodeMismatch [404]; no traffic sent | Pass |
| Alarm triggers automatic rollback | Alarm fires, deployment reverts | Alarm never left OK; no state change in its history | Fail — and that is the finding |
| Failed deployment gives up on its own | Reverts | Retried for over ten minutes until reverted by hand | Fail — circuit breaker was off |
Target.ResponseCodeMismatch — "Health checks failed with these codes: [404]".Security and Cost
Security
No static AWS credentials. HCP Terraform assumes its role via OIDC.
Tasks reachable only through the ALB. The task security group accepts traffic from the load balancer's security group and nothing else.
Scoped IAM. Two AWS managed policies, one per role, no inline wildcards.
Known gap: tasks run in public subnets with public IPs to avoid NAT costs. Production puts them in private subnets.
Cost
Billed $0.1552 for about three hours, read from Cost Explorer once AWS posted the charges: Elastic Load Balancing $0.1126 and ECS/Fargate $0.0425 across 29–30 September, with $0.00 on the 1st — which is what a verified teardown looks like on a bill.
The load balancer was 73% of it, for a service that served a few hundred requests. That is the shape worth remembering: the meter is the front door, not the containers behind it. About $0.05/hour with both versions running, roughly $16/month for an ALB left up.
Troubleshooting and Lessons
The rollback never happened, and that is the finding
Problem. The deliberately broken version was deployed and the automatic
rollback alarm never fired.
Root cause. The broken version failed its health check
(Target.ResponseCodeMismatch — 404), so ECS never sent it any traffic. No
client request reached it, so there were no errors, so the alarm stayed OK and
rolled back nothing. Alarm history confirms no state change during the deployment.
Resolution. None needed — users were protected. But by the health check,
not by the mechanism built for the job.
Lesson. If you only ever test rollback with a fault your health check
catches, you have tested the health check. Use a fault that passes the health check and
fails for users.
A failing deployment does not give up on its own
Problem. The failed deployment cycled tasks for over ten minutes.
Root cause. deploymentCircuitBreaker is off by
default. The deployment retried rather than reverting.
Resolution. Enable the circuit breaker with rollback.
Lesson. Two separate safety features, both off or inert by default, neither
doing what the name suggests without configuration.
A successful apply is not a successful deployment
Problem. terraform apply returned applied with zero
new tasks running and the old version still serving.
Root cause. Terraform hands off to ECS and returns; the deployment continues
asynchronously.
Resolution. Watch the deployment from the client side, not the apply output.
Lesson. Gate a pipeline on that apply and you have gated on
nothing.
Cleanup and Key Takeaways
Destroyed via HCP, then verified independently of the teardown script: no load balancer, no cluster, no target groups, no leftover network interfaces, workspace at zero resources.
bash scripts/cleanup.sh
# CLEAN - nothing from Week 21 remains.
Everything here bills by the hour, so an incomplete teardown is not a tidy loose end — it is roughly $16/month. The $0.00 on the following day is the evidence that the destroy was complete.
Key takeaways
- ECS does blue/green itself now. No CodeDeploy for new work — one less service in the failure path.
- Test rollback with a fault your health check cannot see. Otherwise you are testing the health check and learning nothing about the alarm.
- Turn on the deployment circuit breaker. Without it a failing deployment retries instead of reverting.
- The load balancer is the cost. It bills hourly with no traffic at all, so tear it down.
- Watch a deployment from the client's side. The console shows task sets; only a caller shows the service.
Comments