Multi-region and disaster recovery

Disaster recovery (DR) is the plan for surviving the loss of a whole data centre or cloud region: power or network failure, a bad config pushed everywhere, a cloud control-plane outage, or deleted data. Running in several availability zones (AZs) inside one region handles most hardware failures; multi-region designs handle the rarer, bigger events.

Two numbers define every DR plan. RPO (recovery point objective) is how much recent data you can afford to lose, for example 'at most 5 seconds of payments'. RTO (recovery time objective) is how long the service may be down, for example 'back within 15 minutes'. A design with RPO = 0 needs synchronous replication across regions, which adds the inter-region round trip to every write.

The common strategies, from cheapest to most expensive: backup and restore (hours of RTO), pilot light (data replicated, minimal compute kept off or tiny), warm standby (a scaled-down copy running, ready to scale up), and multi-site active-active (all regions serve traffic). Active-active has the best RTO but forces you to handle writes in two places at once, which means conflict resolution or partitioning users by home region.

1. 1Agree RPO and RTO per data set and journey with the business. Payments ledger: RPO near 0, RTO 15 minutes. Product catalogue: RPO 1 hour (can be rebuilt), RTO 1 hour. Analytics: RPO 24 hours.
2. 2First get multi-AZ right in one region: replicas in at least 2-3 AZs, load balancers across AZs, no single-AZ dependencies. This handles most real outages at low cost.
3. 3Back up everything to another region, and protect backups from deletion (separate account, object lock). Backups also cover the most common 'disaster': bad code or a human deleting data, which replication would copy instantly.
4. 4Choose replication: asynchronous cross-region replication gives low write latency but loses the last seconds of writes on failover (RPO = replication lag). Synchronous replication or consensus across regions (Spanner, CockroachDB, a 3-region Raft group) gives RPO 0 but every write waits for another region, adding tens of milliseconds.
5. 5Choose the traffic model. Active-passive: one region takes writes, the other is standby. Active-active with home regions: each user or tenant is pinned to one region for writes, so there are no conflicts. Active-active multi-writer: any region writes and conflicts are resolved by last-writer-wins, CRDTs or application logic.
6. 6Route traffic with DNS (health-checked records with short TTLs like 30-60 s), global anycast load balancers, or client-side region lists. Remember that some resolvers and apps ignore low TTLs.
7. 7Make failover safe: promote the standby database only after the old primary is fenced (cannot accept writes), to avoid split brain where both regions take writes and diverge.
8. 8Keep the standby honest: same code version, same config, enough capacity (or pre-approved quota to scale up), and secrets and certificates already present.
9. 9Prefer static stability: the standby should be able to serve without calling control-plane APIs of the failed region, because those are often affected by the same outage.
10. 10Test regularly with game days: fail over a real service, measure actual RTO and RPO, and fix what breaks. Many teams find hard-coded endpoints and missing permissions only during a test.