Homeβ€Ί Blogβ€Ί Week 28 - The Failover Nobody Runs: Twelve Minutes…
AWS Weekly Lab AWS Terraform

The failover nobody runs

Every disaster recovery document contains a recovery time. Almost none of them contain a measurement, because the only way to test a read replica is to destroy it.

Verified against current vendor documentation on 18 November 2026. Pricing, limits and API behaviour were checked against the official docs on that date. Cloud services change fast — if you are reading this much later, treat the specifics as a starting point and re-check the linked sources.
AWS Platform Engineering Lab Β· Week 28 of 52

Lab Overview

A building has a fire drill because walking to the exit is free. You can do it on a Tuesday, twice, and nothing is lost but ten minutes.

Database failover has no equivalent. Promoting a read replica turns it into a standalone database that no longer replicates, and there is no way back. Rehearsing the procedure destroys the thing the procedure protects. So the recovery time in the document is almost always a guess that nobody has ever been able to afford to check.

So this week I ran it: a PostgreSQL primary in Virginia, a cross-region read replica in Oregon, and an AWS Application Recovery Controller Region switch plan to promote it β€” then measured the two numbers nobody can tell you without running it. How far behind the replica was, and how long recovery actually took.

Cloud Services and Concepts

ServicePurpose hereWhat it is
Amazon RDS for PostgreSQLThe primary and its cross-region read replica, both db.t4g.micro running 18.6A managed relational database; a read replica is a second instance fed asynchronously from the first
ARC Region switchThe failover runbook as a plan, executed to promote the replica β€” and the only flat monthly fee hereApplication Recovery Controller's recovery orchestration β€” a plan of workflows and execution blocks
AWS LambdaThe write load, inside the primary's VPC because neither database has a public endpointShort-lived functions; here one writes rows on demand and nothing runs between invocations
Amazon CloudWatchReplicaLag β€” the data-loss window, and the metric most likely to be misreadThe metric store; ReplicaLag is published every 60 seconds without being asked
AWS KMSEncryption at rest; a cross-region replica needs a key in the destination regionManaged encryption keys; keys are regional, which matters for a cross-region replica
HCP TerraformRemote applies through OIDC across two regions, so no AWS keys are stored anywhereHosted Terraform runs, authenticating to AWS with short-lived credentials through OIDC

Cross-region read replica. A second database in another region that receives changes asynchronously. It can serve reads, and it can be promoted to a primary. "Asynchronously" is the whole story: whatever has not arrived when the first region fails is gone.

ARC Region switch. A plan is a recovery runbook expressed as workflows and execution blocks. AWS added RDS orchestration and Terraform support in February 2026, which is what makes the runbook reviewable in a pull request instead of a document that rots.

What ReplicaLag actually measures

This is the trap, and I walked straight into it. For PostgreSQL, AWS documents the metric as exactly this query:

SELECT extract(epoch from now() - pg_last_xact_replay_timestamp())

Read it carefully. now() keeps moving. pg_last_xact_replay_timestamp() does not, because there is nothing to replay. So on an idle primary the metric climbs by sixty seconds every minute while the replica is perfectly caught up, and collapses the instant a write lands.

My first readings were 218, 278, 38, 98, 158, 218, 278, 38. That looks like a database minutes behind. It means the opposite: nobody was writing. A dashboard alarm on that metric fires hardest when the system is doing nothing at all.

Promotion is one-way

The Promote Read Replica execution block supports active/passive plans only, and AWS states plainly that after promotion "the promoted instance will no longer replicate from the original primary database". Failing back is not a button. It means building a new replica in the other direction and promoting that.

Architecture and Workflow

aws AWS Cloud Region: us-east-1 (primary) Region: us-west-2 (standby) writer 6.4M rows PostgreSQL 18.6 db.t4g.micro read replica db.t4g.micro 1 2 asynchronous β€” lag 4.2s avg, 20s peak 3 ARC Region switch plan β€” activate us-west-2 β†’ promote-replica AFTER PROMOTION: standalone primary. There is no un-promote. DATABASE SERVING AGAIN: 160 s after the promote call

The numbers in the diagram are the order things happen:

  1. Write load. A Lambda function in the primary's VPC writes 6.4 million rows, because replication lag is only meaningful while something is actually replicating.
  2. Asynchronous replication to Oregon. This is the only link between the two regions, and the gap it leaves is the data you would lose.
  3. The Region switch plan promotes the replica. It is drawn outside both regions on purpose: ARC runs a data plane in every region, so activating Oregon never calls into the region being evacuated. That is the difference between a recovery tool and a Lambda that dies with the region it lives in.

Prerequisites

  • An AWS account, and Terraform 1.10 or later.
  • The AWS provider at 6.47 or later β€” rds_promote_read_replica_config landed in that release.
  • Backups enabled on the source. A read replica cannot be created from an instance with backup_retention_period = 0. This is a prerequisite, not a durability preference.
  • Two regions with a default VPC, or your own subnets in each.

Implementation

What was deployed

ResourceHow manyIts job
aws_db_instance2The primary in us-east-1 and its cross-region replica in us-west-2, both db.t4g.micro, PostgreSQL 18.6
aws_arcregionswitch_plan1The failover runbook: two activate workflows, one per region
aws_lambda_function1Writes load to the primary so lag is measurable
aws_db_subnet_group2One per region; a replica needs its own
aws_security_group3Database and writer in the primary region, database in the replica region. The replica's has no ingress at all β€” replication arrives over the managed path
aws_iam_role2One the writer runs as, one ARC assumes to promote
Rules, policies, password6Ingress and egress rules, role policies, generated credential

Seventeen resources. Two of them cost real money.

The HCP Terraform run page for workspace week-28-dev, showing the plan finished with 17 resources to create and the apply in progress.
Seventeen to create, applying. The replica alone took 14 minutes to build from its source.
The HCP workspace variables page showing TFC_AWS_PROVIDER_AUTH set to true and TFC_AWS_RUN_ROLE_ARN marked sensitive and write-only.
Two regions, no stored keys. The run assumes a role through OIDC; the role ARN is write-only.

The one thing that is not obvious in the Terraform

A cross-region replica of an encrypted source needs storage_encrypted = true declared on the replica, even though RDS sets it for you. AWS reports true, your config says null, and Terraform reads that as true -> null # forces replacement.

The first apply succeeds either way. The second one destroys the database. That is what makes it a trap rather than an error.

The ARC Region switch plan detail page showing recovery approach Active/passive, two workflows, and the activate us-west-2 workflow containing an Amazon RDS promote read replica execution block named promote-replica.
The runbook, as the console sees it. The same thing lives in version control as about thirty lines of HCL.

Validation and Evidence

Lag first, because it stops existing the moment you fail over.

A CloudWatch graph of ReplicaLag over two hours, showing a tall sawtooth rising to nearly eleven minutes while the primary is idle, then flattening to near zero once sustained writes begin.
The same metric, two stories. The tall sawtooth is an idle primary. The flat stretch is 6.4 million rows being written.

Under sustained load the replica held a 4.2 second average and a 20 second peak, across roughly 2,400 miles. That peak is the honest answer to "how much would we lose" β€” and it is two orders of magnitude smaller than the idle metric advertised.

TestExpectedActualStatus
Replication established cross-regionreplica reports the primary as its sourcesource set, status availableβœ…
ReplicaLag under sustained writesseconds, not minutes20 s peak, 4.2 s averageβœ…
Plan execution, activate us-west-2reaches completedpending β†’ inProgress β†’ completedβœ…
Database serving as standalone primaryminutes160 s (promote call to restart)βœ…
Plan execution reports completesoon after the database is up726 s β€” it waits out a backupβœ…
Endpoint after promotionunchangedunchangedβœ…
Replication after promotiongone, irreversiblysource is nullβœ…

That table has two durations in it, and the difference between them is the most useful thing I found all week. My script waited until the API stopped reporting a replication source, got 726 seconds, and I nearly published that as the recovery time. AWS's own event log, below, says otherwise.

A terminal showing CloudTrail reporting PromoteReadReplica called by the identity RegionSwitchExecution-93de3f35793a4ed6, followed by the RDS event log: shared_buffers reduced under memory pressure at 05:16:02, automated backups enabled at 05:17:02, Promoted Read Replica to a stand-alone database instance at 05:17:03, DB instance shutdown at 05:17:15, DB instance restarted at 05:17:35, backup started at 05:18:05 and finished at 05:26:48.
The record AWS keeps, which outlives the resources. Note the caller: RegionSwitchExecution-…, not a person. The plan did the promotion.

Read the timestamps. PromoteReadReplica was called at 05:14:55. The instance restarted at 05:17:35 β€” 160 seconds later, and from that moment it is a standalone primary accepting writes. What ran until 05:26:48 was an automatic backup, which is RDS tidying up after itself, not an outage.

Two honest answers, then. The database was back in under three minutes; the orchestrator did not call itself finished for twelve. Quote the second as your RTO and you overstate your own recovery fourfold.

The same log records RDS halving shared_buffers at 05:16:02, because the write load pushed a db.t4g.micro critically low on memory. My own instrumentation never noticed.

Security and Cost

Neither database has a public endpoint. The replica's security group has no ingress rules at all, because replication does not travel through one β€” a useful thing to see written down, since the instinct is to open a port for it. The writer reaches the primary through a security group reference rather than a CIDR, and has no route to the internet.

What this means for a platform team

The runbook being Terraform is the point. A recovery document is reviewed when someone remembers; a plan in version control is reviewed when it changes, and it fails CI when it is wrong. Mine failed three times before it was right, which is three times it would have failed during an incident instead.

Cost

$0.2359 measured, read 2026-11-19 UTC once the UTC billing day the lab ran on (18 November) had closed. Two db.t4g.micro instances and their storage came to $0.1604 over about two hours and forty minutes. The Region switch plan came to $0.0755.

That second figure is the one worth having, because the pricing page says "$70 per plan per month" and "prices are prorated for partial months" without saying what partial means. The bill says exactly what it means:

Global-AmazonARCRegionSwitchPlan   0.001078 Months   $0.0755
implied rate: $69.999998 per month

Metered in fractional months, at exactly the published rate. So the fee is real but the granularity is fine: a plan that exists for an hour costs about seven cents, and rehearsing your failover does not cost you a month of plan fees. A plan you keep, which is the point of having one, is $70 a month or $840 a year per application.

One thing I cannot account for: the plan existed for 81 minutes by CloudTrail's record, and the meter recorded 0.001078 months, which is about 47. I am reporting both rather than guessing at the difference.

Troubleshooting and Lessons

Problem. The apply failed on the Region switch plan with ValidationException: activePassive plans must not specify target action 'deactivate'. Removing the deactivate workflow produced a second refusal: "must have activate workflows defined for each region but found workflow for only us-west-2".

Root cause. The Terraform provider's own documented example shows an activate workflow paired with a deactivate workflow. That shape is not valid for an active/passive plan, which wants an activate workflow for every region in the plan.

Resolution. Two activate workflows: one targeting us-west-2 containing the promote block, one targeting us-east-1 containing a manual approval, because activating the original region after a promotion is not something that can be automated.

Lesson. A provider example is not an API contract. This one cost two applies to discover, and both error messages were precise once read as specifications rather than complaints.

Problem. The plan reported "Action required" in both regions, with a missing-policy warning and, separately, "Region switch does not have data on DB instance … validate that the plan execution role has sufficient IAM permissions".

Root cause. One missing permission, iam:SimulatePrincipalPolicy, which is not in AWS's published sample policy for the RDS execution block. ARC pre-flights the execution role against each step; when it cannot simulate the role, it reports both the missing policy and its inability to see the database.

Resolution. Grant iam:SimulatePrincipalPolicy, and arc-region-switch:ApprovePlanExecutionStep for the manual approval step.

Lesson. Two warnings that read like unrelated faults can share one cause. Plan evaluation is also cached and does not re-run promptly, so this is slow to iterate β€” worth getting right from the documentation rather than by trial.

Problem. StartPlanExecution returned "An activate flow should be run on the target region. Please start the execution in the target region: us-west-2".

Root cause. The call was made against the primary region, on the assumption that a plan is addressed where it was created.

Resolution. Create the client in the region being activated.

Lesson. This is the per-region data plane working as designed. If activating Oregon required a call into Virginia, the plan would depend on the region it exists to evacuate β€” which is the failure mode a hand-rolled promotion Lambda has and nobody notices until the day it matters.

Problem. The first successful apply still errored: "Provider produced inconsistent result after apply … block count changed from 0 to 1", and every later plan proposed swapping the two workflows.

Root cause. The provider compares workflows by position, while GetPlan returns them sorted by region. The config declared us-west-2 first, so position 1 in the config and position 1 in the response were different workflows.

Resolution. Declare the primary region's workflow first, matching the order the API returns. The plan then shows no changes.

Lesson. The resource was created correctly in AWS despite the error β€” worth checking the service before assuming a failed apply means nothing happened, particularly for a resource billed as a flat monthly fee.

Cleanup and Key Takeaways

A terminal querying both regions for week28 database instances, manual snapshots and Region switch plans. Every query returns an empty list.
Both regions, instances, snapshots and the plan. Snapshots are included because they outlive a destroyed instance and keep charging.

Key takeaways

  • ReplicaLag on an idle primary measures idleness. It is now() minus the last replayed transaction, so it climbs forever when nothing is being written. Read it under load or do not read it.
  • Measure the database, not the orchestrator. The plan reported 726 seconds; the database was serving in 160. The gap is an automatic backup. Check the RDS event log rather than trusting whatever your own script decided to wait for.
  • A documented RTO is a guess until someone runs it. This one was 160 seconds to a serving database. Yours is a different number, and you do not know it.
  • Promotion is irreversible, which is exactly why the drill never happens. Budget for rebuilding the replica afterwards, because that is the real cost of testing.
  • Put the runbook in version control. Three API rejections found three mistakes in mine. An incident is a worse place to find them.

References

Comments

How was your experience?
Your feedback helps improve this site.
PoorExcellent