Homeβ€Ί Blogβ€Ί AWS Architecture Series #120 β€” Recovery is a migration that looks like a reboot, and only one check starts it…
AWS Architecture AWS Architecture Series

AWS Architecture Series #120 β€” Recovery is a migration that looks like a reboot, and only one check starts it

EC2 has two automatic recovery mechanisms, one of them on by default, and both are started by exactly one signal: a failed system status check. Automatic instance recovery does not operate if only the instance status check fails. That narrowness is correct, but it collides with something that changed this week - a failed performance status check now also increments the aggregate StatusCheckFailed metric. A recover action wired to the aggregate rather than to StatusCheckFailed_System will migrate an instance for a degraded GPU whose documented remediation is a reboot. And neither mechanism can be retrofitted: both must be configured in advance, before a system status check fails.

Verified against current vendor documentation on 21 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.

Business Challenge

Two days ago EC2 gained a fifth status check, for accelerated instances that are degraded but reachable, and the documentation noted in passing that a failed performance status check “also increments the StatusCheckFailed metric”. That is the aggregate. The obvious next question is what, exactly, is wired to the aggregate — and for a great many accounts the answer is an instance recovery action written years ago.

So this is worth getting precise about, because recovery is one of those features that everybody has switched on and very few people have read. The usual mental model is if my instance breaks, AWS restarts it somewhere else. Each clause of that is slightly wrong. It is not any break, it is not a restart, and whether it happens at all was decided when the instance launched.

Three facts decide whether recovery helps you, and none of them are intuitive.

1There are two mechanisms, and the default one excludes the least replaceable instances

“If AWS detects that an instance is unavailable due to an underlying hardware or software issue, there are two mechanisms that can automatically restore instance availability” — simplified automatic recovery and CloudWatch action based recovery.

Simplified recovery is Enabled by default on supported instances, which is why most teams have it without having chosen it. But it is Excluded for metal instance sizes and Not supported for instances that attach instance store volumes at launch — precisely the shapes nobody wants to lose. Those instances have no recovery at all unless somebody built a CloudWatch alarm, and nothing announces the absence.

2One status check out of five starts a recovery, and the aggregate metric now has five feeds

“Automatic instance recovery can recover an instance that fails a system status check. However, automatic instance recovery does not operate if only the instance status check fails.”

The correct signal for a recover action is StatusCheckFailed_System. The aggregate StatusCheckFailed is a different thing, and it is a different thing than it was last week, because “A failed performance status check also increments the StatusCheckFailed metric.” An alarm written years ago for a host failure can now migrate an instance for a degraded GPU.

3Recovery is a migration that presents as a reboot, and it keeps what a manual fix throws away

“During the instance recovery process, AWS will attempt to move your instance from the host with the underlying hardware or software issue to a different host. If successful, the instance recovery process will appear to the instance as an unplanned reboot.”

RAM is gone and uptime resets, but the instance ID and the addresses survive — and the documentation draws the contrast with the thing an engineer does by hand: “unlike automatic instance recovery, where the instance retains its public IPv4 address, a restarted instance receives a new public IPv4 address unless it has an Elastic IP address.” The managed path is gentler than the manual one, which inverts the usual instinct.

Architecture

Diagram: Amazon EC2 automatic instance recovery. If AWS detects that an instance is unavailable due to an underlying hardware or software issue, there are two mechanisms that can automatically restore instance availability: simplified automatic recovery, enabled by default during instance launch with fixed AWS-managed behaviour, excluding metal instance sizes, not supported for instances that attach instance store volumes at launch, notifying only through the Health Dashboard and carrying no additional cost; and Amazon CloudWatch action based recovery, which requires manual configuration of CloudWatch alarms and actions, allows customisable actions and conditions and SNS notification, includes metal sizes, is supported on selected instance types where instance store data is lost during recovery, makes faster recovery attempts, and might incur CloudWatch charges. Both must be configured in advance before a system status check fails. The trigger is narrow: if a system status check fails AWS initiates automatic instance recovery, which attempts to migrate the affected instance to different hardware, and automatic instance recovery does not operate if only the instance status check fails. A failed system status check indicates a problem with the host hardware or software and not a problem with the instance itself, from causes including loss of network connectivity, loss of system power, software issues on the physical host, and hardware issues on the physical host that impact network reachability. The metric to alarm on is StatusCheckFailed_System, reporting 0 when the check passed and 1 when it failed. The trap is that a failed performance status check also increments the aggregate StatusCheckFailed metric, so a recover action wired to the aggregate can migrate an instance for a degraded GPU whose documented remediation is a reboot. Recovery is a migration that presents as an unplanned reboot. A recovered instance is identical to the original except for what is lost: preserved are the instance ID, public, private and Elastic IP addresses, instance metadata, placement group, attached EBS volumes and Availability Zone; lost are data in volatile memory, instance store data for CloudWatch action based recovery only, and operating system uptime which resets to zero. The instance retains its public IPv4 address during recovery, whereas a manually restarted instance receives a new public IPv4 address unless it has an Elastic IP address. If recovery is unsuccessful the instance might continue running on the impaired host and manual intervention is required. Recovery attempts emit Health Dashboard events per mechanism. These mechanisms are designed for individual instances.

Two mechanisms, and only one of them is on by default

“If AWS detects that an instance is unavailable due to an underlying hardware or software issue, there are two mechanisms that can automatically restore instance availability” — simplified automatic recovery, and CloudWatch action based recovery.

Most teams have the first without knowing it and believe they have the second. They are not interchangeable, and the differences land precisely on the fleets where recovery matters most:

Comparison pointSimplified automatic recoveryCloudWatch action based recovery
ConfigurationEnabled by default on supported instancesRequires manual configuration of CloudWatch alarms and actions
FlexibilityFixed recovery behavior managed by AWSCustomizable actions and conditions
NotificationHealth DashboardCustomizable through SNS
Metal instance sizeExcludedIncluded
Instance store volumes attached at launchNot supportedSupported on selected types — and the data is lost
Recovery timeStandard recovery attemptFaster recovery attempts
CostNo additional costMight incur CloudWatch charges

Read the metal row and the instance-store row together and a pattern appears: the default mechanism excludes exactly the shapes that cannot be casually replaced. A metal host, or an instance whose local NVMe holds working state, gets no simplified recovery at all. Those are the instances a team is least willing to lose and most likely to assume are covered.

And the retrofit does not exist: “To benefit from the automatic recovery mechanisms, they must be configured in advance on an instance before a system status check fails.” Reacting to an impairment by enabling recovery is too late by definition.

One check out of five starts it

“If a system status check fails, AWS initiates automatic instance recovery, which attempts to migrate the affected instance to different hardware.” And the exclusion is stated just as plainly: “automatic instance recovery does not operate if only the instance status check fails”.

That is the whole trigger surface. EC2 now publishes “five types of status checks: system, instance, attached EBS, application, and performance”, and recovery responds to one of them. The reasoning is sound rather than arbitrary: “A failed system status check indicates a problem with the host's hardware or software, and not a problem with the instance itself”. Moving a guest to a different host only helps when the host is the problem. A kernel panic travels with you.

The documented causes are all host-level: “loss of network connectivity, loss of system power, software issues on the physical host, and hardware issues on the physical host that impact network reachability”.

The metric to alarm on, and the one to stop alarming on

For CloudWatch action based recovery the correct signal is named in the documentation: “The StatusCheckFailed_System metric in CloudWatch indicates whether a system status check passed or failed”, reporting 0 when it passed and 1 when it failed.

The aggregate StatusCheckFailed is a different thing, and as of this week it is a materially different thing: “A failed performance status check also increments the StatusCheckFailed metric.”

So a recover action attached to the aggregate now fires for a condition that recovery is the wrong answer to. A degraded GPU is not a host failure; its documented remedy is a reboot to reset the GPU and trigger row remapping, which resolves the issue without replacing the instance. Migrating instead is slower, discards the fix AWS actually recommends, and on accelerated capacity it trades a short reboot for a relaunch into a pool that may not have room.

One alarm, one metric, one remedy. The aggregate is a reasonable thing to put on a dashboard for a human to interpret; it is a poor thing to attach an automated action to, because it unions conditions whose correct responses differ.

What survives a recovery

“A recovered instance is identical to the original instance, except for the elements that are lost.” Preserved: instance ID, public, private and Elastic IP addresses, instance metadata, placement group, attached EBS volumes, Availability Zone.

Lost: “Data stored in volatile memory (RAM) will be lost and the operating system's uptime will start over from zero. Furthermore, with CloudWatch action based recovery, data on instance store volumes will also be lost.”

The instance-ID and IP preservation is the operationally valuable part, and it is what separates recovery from the manual fix people reach for. The documentation makes the contrast explicitly: “unlike automatic instance recovery, where the instance retains its public IPv4 address, a restarted instance receives a new public IPv4 address unless it has an Elastic IP address”.

That is worth sitting with. A stop/start — the thing an engineer does at 3am when an instance is unreachable — silently changes the instance's public address. Anything holding that address (a DNS record, a partner's allow-list, a hard-coded config) breaks, and it breaks as a consequence of the fix rather than of the fault. Recovery does not do that. Letting the mechanism act before intervening by hand is therefore the lower-blast-radius path, which inverts the usual instinct.

The uptime reset deserves a note too: anything that keys off uptime — a licence check, a cron-style scheduler, a heartbeat that treats low uptime as a fresh boot, a monitoring rule that suppresses alerts for newly booted hosts — sees a brand-new machine. The instance ID did not change, so inventory reconciliation will not flag it.

It can fail, and it can change its mind

Recovery is an attempt, not a guarantee: “If the recovery process is unsuccessful, the instance might continue running on the host with the underlying hardware or software issue. In this case, manual intervention is required.”

And in both mechanisms, “Migration might be canceled and the instance stays on the original host” if the host problem resolves mid-migration. Which means an impairment can appear, a recovery can begin, and the instance can end up exactly where it started with no reboot at all. A runbook that assumes impairment implies reboot will mis-explain what happened.

This is where the Health events matter. Recovery attempts emit distinct Health Dashboard event types per mechanism and per outcome — success and failure variants for simplified recovery, and a separate pair for CloudWatch action based recovery. That is the audit trail for whether recovery fired and whether it worked, and it is the same Health event channel that just gained resource and account tags, so these events can now carry the tags of the instance they are about.

Why This Architecture Holds Up

Restricting the trigger to system checks is a feature

A recovery that fired on any impairment would be actively harmful. Instance status check failures are guest problems, and migrating a guest carries its problem along while costing an unplanned reboot. Narrowing the trigger to the one class of fault that changing hosts can actually fix is the correct design, and AWS states the reasoning rather than leaving it implicit.

Enabling simplified recovery by default is the right call

The failure mode of a default-off resilience feature is that it is off on the instance that needed it, discovered during the incident. Making it the launch default, at no additional cost, with fixed AWS-managed behaviour, means the common case is covered without anyone deciding anything. The exclusions — metal, instance store at launch — are the shapes where a generic AWS-managed action genuinely cannot be safe.

Presenting as an unplanned reboot is the honest abstraction

AWS could have described recovery as transparent. Instead the documentation says it “will appear to the instance as an unplanned reboot” and spells out that RAM is gone and uptime resets. That is what the guest actually experiences, and saying so lets you reason about it: anything that cannot survive an unplanned reboot cannot survive a recovery either, so the test is one you already know how to run.

And the admission that this is not a resilience strategy

“Automatic instance recovery mechanisms are designed for individual instances.” AWS follows it by recommending load balancing and Auto Scaling for a resilient system. A feature whose documentation tells you it is not the architecture you want is being straight with you, and it is the line to quote when someone proposes recovery as the availability plan.

Key Architecture Decisions

DecisionTakeWhy
Which mechanism Leave simplified on everywhere; add CloudWatch action based recovery on metal and instance-store-at-launch instances Those two shapes are excluded from or unsupported by the default, and they are the least replaceable
Which metric for a recover action StatusCheckFailed_System, never the aggregate The aggregate now includes a performance condition whose documented remedy is a reboot, not a migration
Instance store workloads Treat local disk as lost on recovery and design for it Instance store data is lost under CloudWatch action based recovery
Manual stop/start during an incident Let recovery act first; stop/start is the escalation, not the reflex A restart changes the public IPv4 address unless there is an Elastic IP; recovery keeps it
Treating recovery as HA Do not It is documented as being for individual instances; the resilient-system answer is ELB and Auto Scaling

The audit worth running

Three questions, and none of them are answerable from the instance:

  • Which instances have neither mechanism? Metal and instance-store-at-launch instances have no simplified recovery, so unless someone built an alarm they have none at all.
  • Which recover actions point at the aggregate metric rather than at StatusCheckFailed_System? That is the alarm whose behaviour changed this week without anyone editing it.
  • Has recovery ever actually fired, and did it succeed? The Health events are the record — the success and failure event types differ by mechanism, so the query also tells you which mechanism acted.

Closing Thought

The interesting thing about recovery is how much of it is decided before anything goes wrong. Which mechanism you have was set at launch. Whether the mechanism covers your instance shape was set by AWS. What the recovery preserves was set by its design. By the time the system status check fails, every meaningful decision has already been made, and the only remaining variable is whether the migration succeeds.

Which is why the audit above is the whole of the work. There is no runbook step for recovery, no lever to pull during the incident, nothing to tune. There is only the question of whether the instances you care about are covered, and whether the alarm you wrote for a host failure is still only watching host failures — because that one stopped being true this week, quietly, in a metric nobody edited.

And the small counter-intuition worth keeping: the instinct to stop and start an unreachable instance is the one action that throws away an address other systems may depend on. The managed mechanism is gentler than the manual fix. That is unusual enough to be worth remembering at 3am.

Next in this series

Reliability — Amazon EC2 Auto Scaling health checks: how an Auto Scaling group decides an instance is unhealthy, why that decision is separate from EC2's own status checks, what the health check grace period actually protects, and how replacement interacts with an instance that recovery would have fixed.

Official AWS Reference

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent