Multi-region design, durability and disaster recovery

Why it matters

Senior design discussions often end with "and what if the whole region goes away?" Good answers are quantitative: what availability the design reaches, how much data could be lost, how long recovery takes and what all of it costs. Weak answers say "multi-region active-active" without noticing the latency, consistency and operational price.

Core ideas

Availability arithmetic. Serial dependencies multiply: three components at 99.9% give about 99.7%. Independent redundant copies combine as one minus the product of their unavailabilities: two 99% replicas give 99.99%, if failures are independent. They rarely are; shared deployments, configuration, power, networks and dependencies correlate failures, so treat the redundancy formula as an upper bound. Convert targets to downtime (99.9% is about 43 minutes per 30 days) to make them concrete.

Durability is not availability. Durability is the probability that stored data is not lost; availability is whether you can reach it now. Data can be durable yet unreachable during a regional outage, or reachable yet silently corrupted.

RPO and RTO. The recovery point objective is how much recent data you can afford to lose, set by replication lag or backup frequency. The recovery time objective is how long restoration may take. Strategies trade cost for both: backup and restore (cheapest, slowest), pilot light (data replicated, compute started on demand), warm standby (a scaled-down live copy) and active-active (fastest recovery, highest cost and complexity).

Cross-region replication. Synchronous replication across regions gives an RPO near zero but adds a wide-area round trip to every write; asynchronous replication keeps writes fast but loses whatever had not replicated at failover. Active-active writes need conflict resolution, a home region per record, or a globally consistent database that pays the latency.

Replicas are not backups. Replication copies deletes, bad migrations and corrupted writes immediately. Backups provide point-in-time recovery, must be isolated from the systems and credentials that could destroy them, and only count if restores are tested regularly.

Static stability. A good zone or region plan needs no new actions during the failure: capacity is already provisioned, and the data plane does not depend on a control plane that may itself be impaired.

Common misconceptions

• "Two regions double our nines." Only for independent failures; a global configuration push or a shared dependency takes down both.
• "We replicate, so we have backups." Replication propagates mistakes; you still need versioned, isolated, restore-tested backups.
• "Active-active is always best." It costs latency or conflict handling and makes every change a multi-region change; many services are better served by a warm standby.
• "Our failover works." Untested failovers and restores tend to fail when needed; schedule drills and measure the achieved RTO.

How to approach it

Practice the arithmetic until it is automatic, then read SRE book Ch. 26 and the AWS DR whitepaper. For a service you know, set RPO and RTO targets, pick a strategy and write what happens, minute by minute, during a regional failure. Senior candidates are expected to discuss cost, correlated failures, data-consistency consequences and the drills that prove the plan.