Business Challenge
A financial services platform processes transactions in us-east-1. The compliance team requires a documented RTO of under 5 minutes and RPO of under 1 minute for the primary database tier. The platform team sets up Aurora Global Database with a secondary cluster in eu-west-1, runs a DR drill, promotes the secondary in 47 seconds, and files the compliance report. The architecture works.
Six months later, us-east-1 suffers a real 3-hour outage. Under pressure the team reaches for the procedure they rehearsed: detach eu-west-1 from the global cluster and promote it. Transactions resume, the RTO target is met. Then the primary region recovers — and nobody knows what to do next. Detaching is what made eu-west-1 a standalone cluster, so there is nothing left to rejoin: the old primary comes back as its own cluster with a write history that diverged the moment the detach happened. There is no automatic merge and no automatic rejoin. The global cluster stays broken until someone rebuilds it by hand.
The uncomfortable part is that this was avoidable, and not by better runbook discipline. AWS offers two unplanned-failover methods. Managed failover — failover-global-cluster --allow-data-loss — promotes the secondary and then, as soon as the old Region is healthy again, adds it back to the global cluster as a secondary automatically, preserving the topology. Manual failover, the detach-and-promote path, is the one that leaves you rebuilding. AWS recommends managed failover for disaster recovery and positions manual failover as the fallback for when it is unavailable — most commonly because the primary and secondary are running engine versions whose patch levels are not compatible.
The DR plan covered failover. It did not cover failback. This is the gap that makes Aurora Global Database implementations incomplete in most enterprises.
Most teams write the failover runbook and consider the job done. Even on the managed path, failback is a deliberate act: Aurora restores the topology for you, but returning the primary role to the original Region is a switchover you choose to run, at a time you choose. On the manual path it is a full rebuild. Either way it needs its own documented procedure, and the first thing that procedure should establish is which of the two paths your engine versions actually leave you on.
Architecture
Aurora Global Database replicates at the storage layer, not the SQL layer. The primary region maintains a writer instance and optional local readers. Up to five secondary regions each maintain a read-only cluster. Replication is asynchronous but fast — typical lag is under one second because it operates on physical redo log segments, not logical SQL statements.
Endpoint strategy
Applications must never hard-code Aurora cluster endpoints. Instead, Route 53 aliases sit in front of the cluster endpoints. During normal operation, write.db.internal points to the primary cluster writer endpoint. During failover, you update the Route 53 record to point to the promoted secondary. Applications reconnect to the same DNS name — no application config change required.
Read traffic uses a separate read.db.internal record with latency-based routing. EU users are served by the eu-west-1 reader cluster; US users by the us-east-1 reader endpoint. This gives you read scalability and geographic latency reduction without any application awareness of the global topology.
Write forwarding
Aurora Global Database supports write forwarding on secondary clusters. When enabled, write statements issued against the secondary are transparently forwarded to the primary writer. This simplifies application architecture — a single connection string works for both reads and writes in any region — but adds latency to writes from the secondary region (round-trip to primary plus replication lag back). Write forwarding is appropriate for low-volume, non-latency-sensitive writes from the secondary region, not for high-throughput write paths.
Why This Architecture Holds Up
Storage-level replication beats logical replication
Aurora replicates redo log segments, not SQL statements. This means replication lag is measured in milliseconds regardless of transaction size, and the secondary does not re-execute queries. A 10GB bulk load that takes 2 minutes in the primary is visible in the secondary within 1 second of each storage write completing.
Secondary clusters serve real read traffic
This is not a cold standby. The secondary cluster actively handles production read queries with local latency. EU users reading account balances get sub-10ms query times from eu-west-1 instead of crossing the Atlantic to us-east-1. The DR capacity is earning its keep every day.
Planned switchover has zero data loss
When you initiate a planned switchover, Aurora drains in-flight writes from the primary before promoting the secondary. The replication lag reaches zero before the role switch completes. The old primary becomes a reader. No data is lost, no reconciliation is needed, and the global cluster remains intact.
RTO in minutes, not hours
AWS puts Aurora Global Database RTO “in the order of minutes” and RPO “typically measured in seconds”. For a managed failover the docs say the chosen secondary typically assumes the primary role within a few minutes — treat sub-minute promotion as a drill result you have measured yourself, not as a number AWS commits to. Note also that after promotion Aurora rebuilds every other secondary, and that rebuild can take from a few minutes to several hours depending on volume size and Region distance.
Key Architecture Decisions
Many teams run the secondary at a smaller instance class to save cost — a db.r6g.large secondary behind a db.r6g.2xlarge primary. This works until failover happens and the secondary is suddenly handling full production write throughput on an undersized instance. The secondary must match the primary's write capacity, not its read capacity.
Size the secondary writer at the same instance class as the primary writer. You can run fewer reader instances in the secondary to reduce cost, but the promoted writer must be able to absorb peak write load immediately.
Which runbook you need depends on the path. After a managed failover Aurora monitors the old primary Region and adds it back as a secondary once it is healthy, rebuilding its storage volume; it also attempts a snapshot of the old volume at the point of failure, named rds:unplanned-global-failover-<cluster>-<timestamp>, which is where any unreplicated writes are recoverable from. Failback is then a switchover, run when you choose. After a manual failover there is no topology to restore: you add Regions to the new primary to rebuild the global database, then switch over.
Write the runbook for the path your engine versions leave you on, and check that assumption on a schedule rather than once. Note the snapshot is a system snapshot subject to the old cluster's backup retention period — copy it to a manual snapshot if you need it to outlive that window.
When the writer endpoint changes during failover, all application connections drop simultaneously. If the application opens connections directly to the Aurora endpoint, a few hundred application instances attempting to reconnect at the same moment can overwhelm the promoted writer before it has finished stabilising.
correct approachPlace RDS Proxy in front of the writer in each region. Proxy maintains a persistent connection pool to Aurora and presents a stable endpoint to applications. During failover, Proxy re-establishes its pool connections to the new writer without the application layer experiencing a mass reconnect storm.
Some teams try to fully automate failover — CloudWatch alarm triggers Lambda which calls the promotion API and updates Route 53. This creates false-positive failover risk: a brief CloudWatch metrics gap triggers promotion unnecessarily, and now you have a broken global cluster to repair from a non-event.
correct approachKeep the promotion decision human. Automate the Route 53 record update and notification only. An on-call engineer confirms the regional failure is real before initiating promotion. Promotion itself is a few minutes — the decision confirmation is usually the longer half.
Tradeoffs
| Decision | Benefit | Cost / Risk |
|---|---|---|
| Aurora Global Database vs Multi-AZ | Cross-region DR, read scaling, <1s RPO | 2× instance cost, failback complexity |
| Write forwarding enabled | Single connection string, simpler app logic | Added write latency (cross-region round trip) |
| RDS Proxy in front of writer | Connection pooling, cleaner failover | Additional cost (~$0.015/hr per AZ), extra hop |
| Planned switchover for maintenance | Zero data loss, global cluster stays intact | Requires coordination, brief write interruption |
| Automated promotion on alarm | Faster recovery, no human in the loop | False positives cause unnecessary failover and data reconciliation work |
Terraform Pattern
The key resource is aws_rds_global_cluster — this is the global cluster object that the primary and secondary regional clusters attach to.
# Global cluster object (region-agnostic)
resource "aws_rds_global_cluster" "main" {
global_cluster_identifier = "platform-global"
engine = "aurora-postgresql"
engine_version = "16.4"
database_name = "platform"
storage_encrypted = true
}
# Primary cluster — us-east-1
resource "aws_rds_cluster" "primary" {
provider = aws.us_east_1
cluster_identifier = "platform-primary"
engine = aws_rds_global_cluster.main.engine
engine_version = aws_rds_global_cluster.main.engine_version
global_cluster_identifier = aws_rds_global_cluster.main.id
database_name = "platform"
master_username = var.db_username
master_password = var.db_password
db_subnet_group_name = aws_db_subnet_group.primary.name
vpc_security_group_ids = [aws_security_group.aurora_primary.id]
kms_key_id = aws_kms_key.aurora_primary.arn
storage_encrypted = true
skip_final_snapshot = false
final_snapshot_identifier = "platform-primary-final"
}
resource "aws_rds_cluster_instance" "primary_writer" {
provider = aws.us_east_1
identifier = "platform-primary-writer"
cluster_identifier = aws_rds_cluster.primary.id
instance_class = "db.r6g.2xlarge"
engine = aws_rds_cluster.primary.engine
engine_version = aws_rds_cluster.primary.engine_version
publicly_accessible = false
}
# Secondary cluster — eu-west-1
resource "aws_rds_cluster" "secondary" {
provider = aws.eu_west_1
cluster_identifier = "platform-secondary"
engine = aws_rds_global_cluster.main.engine
engine_version = aws_rds_global_cluster.main.engine_version
global_cluster_identifier = aws_rds_global_cluster.main.id
db_subnet_group_name = aws_db_subnet_group.secondary.name
vpc_security_group_ids = [aws_security_group.aurora_secondary.id]
kms_key_id = aws_kms_key.aurora_secondary.arn
storage_encrypted = true
skip_final_snapshot = false
final_snapshot_identifier = "platform-secondary-final"
depends_on = [aws_rds_cluster_instance.primary_writer]
}
resource "aws_rds_cluster_instance" "secondary_reader" {
provider = aws.eu_west_1
identifier = "platform-secondary-reader"
cluster_identifier = aws_rds_cluster.secondary.id
instance_class = "db.r6g.2xlarge" # match primary writer size
engine = aws_rds_cluster.secondary.engine
engine_version = aws_rds_cluster.secondary.engine_version
publicly_accessible = false
}
# Route 53 write endpoint alias
resource "aws_route53_record" "db_write" {
zone_id = var.private_zone_id
name = "write.db"
type = "CNAME"
ttl = 30
records = [aws_rds_cluster.primary.endpoint]
}
Closing Thought
Aurora Global Database solves the hard part of multi-region database architecture: keeping a secondary region current enough to take over production traffic within seconds, not hours. The replication lag target of under one second is not a marketing claim — it reflects the architecture's use of physical storage replication rather than logical query replay.
But the architecture only delivers its promise when the team has practised the full lifecycle. Failover without a tested failback plan is a trap: the drill passes, the compliance report is filed, and the real incident reveals a gap that takes days to close. Build both runbooks, test both paths, and size the secondary to absorb writes from day one — not from the moment a region goes down.
Most organisations look at their AWS bill once a month, after the fact. We look at how to build a self-service cost intelligence platform using Cost and Usage Report 2.0 and Athena that gives engineering teams real-time visibility into where spend is going and why.
Official AWS Reference
- Using Amazon Aurora Global Database — topology, advantages and limitations
- Using switchover or failover in Aurora Global Database — the managed and manual failover paths, and what each means for failback
- Removing a cluster from an Amazon Aurora global database
- Using write forwarding in an Amazon Aurora global database
- Monitoring an Amazon Aurora global database — AuroraGlobalDBRPOLag and AuroraGlobalDBReplicationLag
Comments