Business Challenge
A B2B SaaS company serves its public API from 240 m5.2xlarge instances spread across three Availability Zones in us-east-1. The fleet scales between 180 and 400 instances on a weekday curve. Every instance is x86. Every instance is On-Demand. Every instance is billed at list price.
Finance flagged EC2 as the largest single line item on the bill, and the platform team already
knows the textbook answer is Spot. They also already tried it. Eighteen months ago they added
four instance types to the Auto Scaling group, set the Spot allocation strategy to
lowest-price, and moved 70% of the fleet onto Spot. A regional demand surge
reclaimed most of that capacity inside ten minutes. The group tried to backfill with
On-Demand in the same instance types and could not get it. The incident review ended with an
informal rule: no Spot on anything a customer touches.
That rule is the expensive part. The outage was real, but the diagnosis was wrong — and so was the remedy. Five specific configuration choices caused it, and every one of them is documented as an anti-pattern.
A Spot capacity pool is one instance type in one Availability Zone. Four types across three AZs is twelve pools — and correlated ones, since they were all from the same generation and family. AWS's stated rule of thumb is to be flexible across at least ten instance types per workload.
FixDescribe the shape of the compute instead of naming instances, so the group draws from dozens of pools.
AWS does not hedge on this one: "We don't recommend the lowest-price
strategy because it has the highest risk of interruption for your Spot Instances."
The cheapest pool is frequently the thinnest pool, so the group deliberately concentrated
itself in the capacity most likely to be reclaimed.
Use price-capacity-optimized, which weighs available capacity and price together.
70% Spot with no guaranteed On-Demand meant a bad enough Spot event could take the serving tier below the minimum needed to carry traffic. Nothing in the configuration reserved a baseline that Spot conditions could not touch.
FixSet OnDemandBaseCapacity to the instance count that must exist regardless of Spot availability.
Without Capacity Rebalancing, Auto Scaling does not replace a Spot instance until after the Spot service has interrupted it and its health check has failed. The group spent the surge permanently behind the interruptions rather than ahead of them.
FixEnable Capacity Rebalancing to act on the rebalance recommendation, which arrives ahead of the two-minute notice.
This is the counterintuitive one. AWS explicitly discourages failing over to On-Demand to handle interruptions, because "failing over to On-Demand Instances can inadvertently drive interruptions for your other Spot Instances" — and once a type-and-AZ combination is under pressure on Spot, On-Demand in that same combination is often unavailable too. The backfill attempt competed with the capacity it was trying to rescue.
FixDiversify wide enough that no single pool matters, rather than building an escape hatch that amplifies the failure.
Architecture
The target is one Auto Scaling group that runs two CPU architectures across two purchase models. Graviton instances carry the fleet when they are available, because their discount compounds with the Spot discount. x86 instances exist mainly to widen the pool count. A fixed slice of On-Demand underneath both guarantees the service survives a total Spot withdrawal.
The mechanism that makes this possible
A Graviton instance needs an arm64 AMI and an x86 instance needs an x86_64 AMI, so a single
launch template cannot serve both. The feature that resolves this is easy to miss: each entry
in the mixed instances policy's Overrides array can carry its own
LaunchTemplateSpecification. The group-level launch template is the default, and
any override that names its own template uses that one instead.
Combine that with attribute-based instance type selection and you get two blocks — one selecting Graviton by CPU manufacturer, one selecting Intel and AMD — each bound to the correct AMI:
{
"AutoScalingGroupName": "api-tier",
"MixedInstancesPolicy": {
"LaunchTemplate": {
"LaunchTemplateSpecification": {
"LaunchTemplateName": "api-tier-arm64",
"Version": "$Latest"
},
"Overrides": [
{
"InstanceRequirements": {
"VCpuCount": { "Min": 4, "Max": 16 },
"MemoryMiB": { "Min": 8192 },
"CpuManufacturers": ["amazon-web-services"],
"BaselinePerformanceFactors": {
"Cpu": { "References": [{ "InstanceFamily": "c6g" }] }
}
}
},
{
"InstanceRequirements": {
"VCpuCount": { "Min": 4, "Max": 16 },
"MemoryMiB": { "Min": 8192 },
"CpuManufacturers": ["intel", "amd"],
"BaselinePerformanceFactors": {
"Cpu": { "References": [{ "InstanceFamily": "c6i" }] }
}
},
"LaunchTemplateSpecification": {
"LaunchTemplateName": "api-tier-x86",
"Version": "$Latest"
}
}
]
},
"InstancesDistribution": {
"OnDemandBaseCapacity": 60,
"OnDemandPercentageAboveBaseCapacity": 20,
"SpotAllocationStrategy": "price-capacity-optimized"
}
},
"CapacityRebalance": true,
"MinSize": 180,
"MaxSize": 400,
"DesiredCapacity": 240,
"VPCZoneIdentifier": "subnet-a,subnet-b,subnet-c"
}
Note what is not in that file: a list of instance types. The group is told the shape of the compute it needs — 4 to 16 vCPUs, at least 8 GiB of memory, CPU performance no worse than the named baseline family — and Auto Scaling resolves that to whatever currently matches. New instance generations are picked up automatically as AWS releases them, which is the main reason attribute-based selection beats a hand-maintained list.
How capacity gets filled
The On-Demand base is satisfied first
OnDemandBaseCapacity: 60 means the first 60 instances are On-Demand regardless of anything else. This is the floor the service is sized to survive on.
Everything above the base splits 20/80
OnDemandPercentageAboveBaseCapacity: 20 applies only above the base. At a desired capacity of 240 that is 60 On-Demand plus 36 more On-Demand and 144 Spot.
Spot is drawn from the deepest cheap pools
price-capacity-optimized selects pools that are simultaneously least likely to be interrupted and lowest priced, across both the arm64 and x86 requirement blocks.
At-risk instances are replaced before they die
Capacity Rebalancing launches a replacement on the rebalance recommendation, waits for it to pass its health check, and only then terminates the at-risk instance.
Why This Architecture Holds Up
Pool count is the actual availability lever
The original design's problem was arithmetic. Four instance types across three AZs is twelve pools. Two attribute blocks that each resolve to twenty-odd instance types across three AZs is well over a hundred, spanning two CPU architectures and three CPU vendors. A surge that drains one family no longer drains the fleet, because the group was never concentrated there. This is why the fix is diversification rather than a failover path — the failover path is what turned a capacity squeeze into an outage.
AWS recommends being flexible across at least ten instance types per workload, and using attribute-based selection specifically because it keeps that flexibility current as new instance types launch.
The two discounts multiply
AWS states that Graviton-based instances cost up to 20% less than comparable x86 instances, and that Spot offers savings of up to 90% against On-Demand. These are independent reductions applied to the same instance-hour, so a Graviton Spot instance is discounted off an already lower base price. That compounding is the reason to put Graviton first in the policy rather than treating it as an equal alternative to x86.
It also changes the migration calculus. Teams often stall on Graviton because rebuilding for arm64 feels like a project with no immediate payoff. Inside a mixed instances group the payoff is immediate and the risk is bounded: if the arm64 image has a problem, those instances fail their health checks and the group fills from the x86 block instead.
Capacity Rebalancing buys minutes instead of seconds
A Spot interruption notice gives you two minutes. The rebalance recommendation is a separate, earlier signal that fires when an instance moves to elevated risk. Acting on the earlier signal is what turns a scramble into an orderly drain — connections finish, the load balancer deregisters the instance, and the replacement is already serving before the old one stops.
Two behaviours are worth knowing before enabling it. Auto Scaling may temporarily exceed the group's maximum size by up to 10% of desired capacity, because it launches before it terminates — a group pinned at its ceiling can otherwise stall rebalancing entirely. And it will only launch a replacement if that replacement has the same or better availability than the instance it replaces, so it will not knowingly trade one at-risk instance for another.
Capacity Rebalancing does not reduce your interruption rate. It acts earlier, which means more total replacements — you are buying graceful continuity, not fewer events. Budget for the extra churn.
Key Architecture Decisions
| Decision | Choice | Reasoning |
|---|---|---|
| Spot allocation strategy | price-capacity-optimized |
AWS's recommended default. Weighs real-time capacity alongside price, so the group avoids thin pools instead of seeking them out. |
| Instance selection | Attribute-based, two blocks | Maximises pool count and adopts new instance generations automatically. A static list goes stale and silently narrows over time. |
| Architecture mix | Graviton block first, x86 second | Graviton's price advantage compounds with the Spot discount. x86 exists to widen the pool count and as a fallback if the arm64 image regresses. |
| On-Demand floor | OnDemandBaseCapacity, not a percentage |
A percentage of a shrinking group also shrinks. An absolute base is a real floor that scaling activity cannot erode. |
| Weak-CPU protection | BaselinePerformanceFactors |
Attribute matching on vCPU count alone will happily select older, slower cores. A baseline family sets a performance floor. |
| Interruption handling | Capacity Rebalancing + lifecycle hook | Acts on the rebalance recommendation rather than the two-minute notice, giving connection draining time to complete. |
Constraints to design around
Several limits in this design are not obvious until you hit them:
Per-override launch templates are CLI/SDK only
The feature that lets one group run both architectures is not available in the console. Build this with the CLI, CloudFormation, or Terraform, and expect the console to render it incompletely.
Four InstanceRequirements blocks, maximum
You cannot express an unlimited taxonomy of workload shapes in one group. Two architectures plus two size envelopes already exhausts the budget.
The console pins On-Demand allocation to lowest-price
In the console the On-Demand allocation strategy is preselected as lowest price and cannot be changed. Choosing prioritized — which matters if you hold Reserved Instances — requires the API.
Lifecycle hooks must finish inside two minutes
AWS is explicit that the custom action has to complete in under two minutes. A drain routine that waits on long-lived connections will be cut off mid-flight.
Price protection is on whether you configure it or not
Attribute-based selection enables price protection by default, and the default On-Demand
threshold is 20% above the price of the lowest-priced current-generation C, M, or R instance
type matching your attributes. For Spot, Auto Scaling applies an optimal threshold
automatically unless you override it. This is usually the behaviour you want, but it means the
group may silently decline instance types you expected it to use. If you are debugging a group
that will not scale out, check price protection before assuming there is no capacity — and
note that you can effectively disable it by setting an absurdly high percentage such as
999999.
The GetInstanceTypesFromInstanceRequirements API returns exactly which instance types your attributes resolve to, without launching anything. Run it against both blocks and count the results — if either returns fewer than ten types, the requirements are too narrow.
Closing Thought
The team in this scenario did not have a Spot problem. They had a concentration problem that Spot exposed. Twelve correlated capacity pools, a strategy that actively sought the thinnest of them, no guaranteed floor, and a failover path that competed with its own fleet — any one of those is enough to produce the outage they had, and none of them is really about Spot.
What makes the mixed instances policy worth the configuration effort is that it turns the question from "which instances do we want?" into "what does this workload actually need, and how many different ways can AWS satisfy it?" Answer the second question broadly enough and the capacity math stops being fragile. The cost reduction is almost a side effect — the fleet gets cheaper because it got more flexible, not the other way round.
The honest caveat: none of this makes Spot suitable for stateful or tightly coupled workloads. AWS says so directly, and no amount of diversification changes it. This pattern works for the stateless serving tier, the CI fleet, and the batch pool. It does not work for the database, and the discipline to keep that line clear is what makes the rest of it safe.
Messaging — SQS, SNS, and EventBridge, and the decision tree enterprises consistently get wrong. Three services with overlapping descriptions, three genuinely different delivery models, and a set of failure modes that only appear once you have picked the wrong one.
Official AWS Reference
- Auto Scaling groups with multiple instance types and purchase options — the mixed instances policy, allocation strategies, attribute-based instance type selection, and the per-override launch templates that allow Graviton and x86 to share a single group.
Comments