π In This Post
Lab Overview
A building has a fire drill because walking to the exit is free. You can do it on a Tuesday, twice, and nothing is lost but ten minutes.
Database failover has no equivalent. Promoting a read replica turns it into a standalone database that no longer replicates, and there is no way back. Rehearsing the procedure destroys the thing the procedure protects. So the recovery time in the document is almost always a guess that nobody has ever been able to afford to check.
So this week I ran it: a PostgreSQL primary in Virginia, a cross-region read replica in Oregon, and an AWS Application Recovery Controller Region switch plan to promote it β then measured the two numbers nobody can tell you without running it. How far behind the replica was, and how long recovery actually took.
Cloud Services and Concepts
| Service | Purpose here | What it is |
|---|---|---|
| Amazon RDS for PostgreSQL | The primary and its cross-region read replica, both db.t4g.micro running 18.6 | A managed relational database; a read replica is a second instance fed asynchronously from the first |
| ARC Region switch | The failover runbook as a plan, executed to promote the replica β and the only flat monthly fee here | Application Recovery Controller's recovery orchestration β a plan of workflows and execution blocks |
| AWS Lambda | The write load, inside the primary's VPC because neither database has a public endpoint | Short-lived functions; here one writes rows on demand and nothing runs between invocations |
| Amazon CloudWatch | ReplicaLag β the data-loss window, and the metric most likely to be misread | The metric store; ReplicaLag is published every 60 seconds without being asked |
| AWS KMS | Encryption at rest; a cross-region replica needs a key in the destination region | Managed encryption keys; keys are regional, which matters for a cross-region replica |
| HCP Terraform | Remote applies through OIDC across two regions, so no AWS keys are stored anywhere | Hosted Terraform runs, authenticating to AWS with short-lived credentials through OIDC |
Cross-region read replica. A second database in another region that receives changes asynchronously. It can serve reads, and it can be promoted to a primary. "Asynchronously" is the whole story: whatever has not arrived when the first region fails is gone.
ARC Region switch. A plan is a recovery runbook expressed as workflows and execution blocks. AWS added RDS orchestration and Terraform support in February 2026, which is what makes the runbook reviewable in a pull request instead of a document that rots.
What ReplicaLag actually measures
This is the trap, and I walked straight into it. For PostgreSQL, AWS documents the metric as exactly this query:
SELECT extract(epoch from now() - pg_last_xact_replay_timestamp())
Read it carefully. now() keeps moving.
pg_last_xact_replay_timestamp() does not, because there is
nothing to replay. So on an idle primary the metric climbs by
sixty seconds every minute while the replica is perfectly caught up, and
collapses the instant a write lands.
My first readings were 218, 278, 38, 98, 158, 218, 278, 38. That looks like a database minutes behind. It means the opposite: nobody was writing. A dashboard alarm on that metric fires hardest when the system is doing nothing at all.
Promotion is one-way
The Promote Read Replica execution block supports active/passive plans only, and AWS states plainly that after promotion "the promoted instance will no longer replicate from the original primary database". Failing back is not a button. It means building a new replica in the other direction and promoting that.
Architecture and Workflow
The numbers in the diagram are the order things happen:
- Write load. A Lambda function in the primary's VPC writes 6.4 million rows, because replication lag is only meaningful while something is actually replicating.
- Asynchronous replication to Oregon. This is the only link between the two regions, and the gap it leaves is the data you would lose.
- The Region switch plan promotes the replica. It is drawn outside both regions on purpose: ARC runs a data plane in every region, so activating Oregon never calls into the region being evacuated. That is the difference between a recovery tool and a Lambda that dies with the region it lives in.
Prerequisites
- An AWS account, and Terraform 1.10 or later.
- The AWS provider at 6.47 or later β
rds_promote_read_replica_configlanded in that release. - Backups enabled on the source. A read replica cannot be created
from an instance with
backup_retention_period = 0. This is a prerequisite, not a durability preference. - Two regions with a default VPC, or your own subnets in each.
Implementation
What was deployed
| Resource | How many | Its job |
|---|---|---|
aws_db_instance | 2 | The primary in us-east-1 and its cross-region replica in us-west-2, both db.t4g.micro, PostgreSQL 18.6 |
aws_arcregionswitch_plan | 1 | The failover runbook: two activate workflows, one per region |
aws_lambda_function | 1 | Writes load to the primary so lag is measurable |
aws_db_subnet_group | 2 | One per region; a replica needs its own |
aws_security_group | 3 | Database and writer in the primary region, database in the replica region. The replica's has no ingress at all β replication arrives over the managed path |
aws_iam_role | 2 | One the writer runs as, one ARC assumes to promote |
| Rules, policies, password | 6 | Ingress and egress rules, role policies, generated credential |
Seventeen resources. Two of them cost real money.
The one thing that is not obvious in the Terraform
A cross-region replica of an encrypted source needs
storage_encrypted = true declared on the replica, even
though RDS sets it for you. AWS reports true, your config
says null, and Terraform reads that as
true -> null # forces replacement.
The first apply succeeds either way. The second one destroys the database. That is what makes it a trap rather than an error.
Validation and Evidence
Lag first, because it stops existing the moment you fail over.
Under sustained load the replica held a 4.2 second average and a 20 second peak, across roughly 2,400 miles. That peak is the honest answer to "how much would we lose" β and it is two orders of magnitude smaller than the idle metric advertised.
| Test | Expected | Actual | Status |
|---|---|---|---|
| Replication established cross-region | replica reports the primary as its source | source set, status available | β |
ReplicaLag under sustained writes | seconds, not minutes | 20 s peak, 4.2 s average | β |
| Plan execution, activate us-west-2 | reaches completed | pending β inProgress β completed | β |
| Database serving as standalone primary | minutes | 160 s (promote call to restart) | β |
| Plan execution reports complete | soon after the database is up | 726 s β it waits out a backup | β |
| Endpoint after promotion | unchanged | unchanged | β |
| Replication after promotion | gone, irreversibly | source is null | β |
That table has two durations in it, and the difference between them is the most useful thing I found all week. My script waited until the API stopped reporting a replication source, got 726 seconds, and I nearly published that as the recovery time. AWS's own event log, below, says otherwise.
RegionSwitchExecution-β¦, not a person. The plan
did the promotion.Read the timestamps. PromoteReadReplica was called at
05:14:55. The instance restarted at 05:17:35 β 160
seconds later, and from that moment it is a standalone primary
accepting writes. What ran until 05:26:48 was an automatic backup, which
is RDS tidying up after itself, not an outage.
Two honest answers, then. The database was back in under three minutes; the orchestrator did not call itself finished for twelve. Quote the second as your RTO and you overstate your own recovery fourfold.
The same log records RDS halving shared_buffers at
05:16:02, because the write load pushed a db.t4g.micro
critically low on memory. My own instrumentation never noticed.
Security and Cost
Neither database has a public endpoint. The replica's security group has no ingress rules at all, because replication does not travel through one β a useful thing to see written down, since the instinct is to open a port for it. The writer reaches the primary through a security group reference rather than a CIDR, and has no route to the internet.
What this means for a platform team
The runbook being Terraform is the point. A recovery document is reviewed when someone remembers; a plan in version control is reviewed when it changes, and it fails CI when it is wrong. Mine failed three times before it was right, which is three times it would have failed during an incident instead.
Cost
$0.2359 measured, read 2026-11-19 UTC once the UTC
billing day the lab ran on (18 November) had closed. Two db.t4g.micro instances and their storage came
to $0.1604 over about two hours and forty minutes. The Region switch plan
came to $0.0755.
That second figure is the one worth having, because the pricing page says "$70 per plan per month" and "prices are prorated for partial months" without saying what partial means. The bill says exactly what it means:
Global-AmazonARCRegionSwitchPlan 0.001078 Months $0.0755
implied rate: $69.999998 per month
Metered in fractional months, at exactly the published rate. So the fee is real but the granularity is fine: a plan that exists for an hour costs about seven cents, and rehearsing your failover does not cost you a month of plan fees. A plan you keep, which is the point of having one, is $70 a month or $840 a year per application.
One thing I cannot account for: the plan existed for 81 minutes by CloudTrail's record, and the meter recorded 0.001078 months, which is about 47. I am reporting both rather than guessing at the difference.
Troubleshooting and Lessons
Problem. The apply failed on the Region switch plan
with ValidationException: activePassive plans must not specify
target action 'deactivate'. Removing the deactivate workflow
produced a second refusal: "must have activate workflows defined for each
region but found workflow for only us-west-2".
Root cause. The Terraform provider's own documented example shows an activate workflow paired with a deactivate workflow. That shape is not valid for an active/passive plan, which wants an activate workflow for every region in the plan.
Resolution. Two activate workflows: one targeting us-west-2 containing the promote block, one targeting us-east-1 containing a manual approval, because activating the original region after a promotion is not something that can be automated.
Lesson. A provider example is not an API contract. This one cost two applies to discover, and both error messages were precise once read as specifications rather than complaints.
Problem. The plan reported "Action required" in both regions, with a missing-policy warning and, separately, "Region switch does not have data on DB instance β¦ validate that the plan execution role has sufficient IAM permissions".
Root cause. One missing permission,
iam:SimulatePrincipalPolicy, which is not in AWS's published
sample policy for the RDS execution block. ARC pre-flights the execution
role against each step; when it cannot simulate the role, it reports both
the missing policy and its inability to see the database.
Resolution. Grant
iam:SimulatePrincipalPolicy, and
arc-region-switch:ApprovePlanExecutionStep for the manual
approval step.
Lesson. Two warnings that read like unrelated faults can share one cause. Plan evaluation is also cached and does not re-run promptly, so this is slow to iterate β worth getting right from the documentation rather than by trial.
Problem. StartPlanExecution returned "An
activate flow should be run on the target region. Please start the
execution in the target region: us-west-2".
Root cause. The call was made against the primary region, on the assumption that a plan is addressed where it was created.
Resolution. Create the client in the region being activated.
Lesson. This is the per-region data plane working as designed. If activating Oregon required a call into Virginia, the plan would depend on the region it exists to evacuate β which is the failure mode a hand-rolled promotion Lambda has and nobody notices until the day it matters.
Problem. The first successful apply still errored: "Provider produced inconsistent result after apply β¦ block count changed from 0 to 1", and every later plan proposed swapping the two workflows.
Root cause. The provider compares workflows by
position, while GetPlan returns them sorted by region. The
config declared us-west-2 first, so position 1 in the config and position
1 in the response were different workflows.
Resolution. Declare the primary region's workflow first, matching the order the API returns. The plan then shows no changes.
Lesson. The resource was created correctly in AWS despite the error β worth checking the service before assuming a failed apply means nothing happened, particularly for a resource billed as a flat monthly fee.
Cleanup and Key Takeaways
Key takeaways
- ReplicaLag on an idle primary measures idleness.
It is
now()minus the last replayed transaction, so it climbs forever when nothing is being written. Read it under load or do not read it. - Measure the database, not the orchestrator. The plan reported 726 seconds; the database was serving in 160. The gap is an automatic backup. Check the RDS event log rather than trusting whatever your own script decided to wait for.
- A documented RTO is a guess until someone runs it. This one was 160 seconds to a serving database. Yours is a different number, and you do not know it.
- Promotion is irreversible, which is exactly why the drill never happens. Budget for rebuilding the replica afterwards, because that is the real cost of testing.
- Put the runbook in version control. Three API rejections found three mistakes in mine. An incident is a worse place to find them.
References
- The full lab guide
- Monitoring read replication β the exact query behind
ReplicaLag - Amazon RDS Promote Read Replica execution block β active/passive only, and what promotion changes
- Region switch in ARC β the per-region data plane
- ARC pricing β $70 per plan per month, and readiness checks closing to new customers
- Amazon RDS execution block sample policy β the four actions, and what it leaves out
- Terraform β
aws_arcregionswitch_plan
Comments