Design a Multi-Region Disaster Recovery Solution — System Design Interview Practice

Interview prompt

Design a multi-region disaster-recovery architecture that meets explicit RPO/RTO targets, replicates data and artifacts, detects regional failure, fails over safely, and supports tested restoration and failback.

• Classify data by replication and recovery requirements, define the write authority, and document RPO, RTO, and acceptable data loss.
• Automate infrastructure and configuration from versioned definitions while fencing an old primary to prevent split-brain writes.
• Replicate databases, objects, secrets, queues, and observability evidence with independent lag and integrity checks.
• Explain health detection, DNS or global routing, degraded mode, restore drills, failback, operator approval, and auditability.

Requirements and scale assumptions

• Provision a secondary region, replicate state and artifacts, monitor lag, declare an incident, fail over, and restore traffic.
• Expose regional health, replication lag, RPO estimate, failover readiness, data validation, and recovery progress.
• Support automatic detection with controlled promotion, operator override, backup restore, failback, and recovery drills.
• Meet RPO below five minutes and a documented RTO for each workload class; do not claim zero loss without synchronous replication.
• Replicate thousands of resources and high-volume data streams without one control-plane queue becoming a failover bottleneck.
• Make failover, promotion, restore, and client retries idempotent and fence stale writers before serving the new region.
• Serve degraded read-only or cached functionality while replication or a dependent region is unavailable.
• Protect 100 services across three regions, with databases, objects, queues, secrets, and infrastructure definitions.
• Partition replication and recovery by service, tenant, data class, and region; isolate high-write workloads.
• Retain backups, replication manifests, restore logs, failover decisions, and drill evidence under retention policy.

• Peak scale: Automate infrastructure provisioning — Capacity assumption that drives partitioning and backpressure.
• Latency target: RPO (Recovery Point Objective) < 5 minutes — User-facing budget for the primary request or read path.
• Durable boundary: Committed before async — The source of truth is Ensure business continuity during outages; Replicate data across multiple regions.
• Async boundary: At-least-once workers — Keep Use Route 53 health checks for failover, RDS Multi-AZ for database HA, S3 Cross-Region Replication for objects off the synchronous path.

Key entities

• ResourceSpecresourceId, tenantId, desiredState, version, policyVersion, updatedAt

  Versioned desired state for a multi region disaster recovery solution managed resource.

• OperationoperationId, resourceId, requestHash, step, attempt, status

  Durable multi region disaster recovery solution reconciliation operation with per-step progress.

• PolicyVersionpolicyId, scope, version, rules, effectiveAt, status

  Auditable multi region disaster recovery solution policy evaluated before provisioning or mutation.

• ReconciliationCheckpointresourceId, provider, observedVersion, cursor, lastError, updatedAt

  Provider-specific multi region disaster recovery solution observation and recovery cursor.

Data flow

1. 1. Accept a desired-state commandThe multi region disaster recovery solution control plane authenticates the tenant, validates policy and quotas, checks the expected version, and records the desired state.
2. 2. Plan a safe operationA planner turns multi region disaster recovery solution desired state into ordered, bounded steps with dependency checks, blast-radius limits, and rollback metadata.
3. 3. Reconcile providers asynchronouslyWorkers apply multi region disaster recovery solution operations through provider adapters, persist checkpoints, rate-limit calls, and treat unknown outcomes as observable state.
4. 4. Publish observed healthThe serving projection joins desired and observed multi region disaster recovery solution state with operation status, policy version, freshness, and actionable errors.
5. 5. Recover and auditRetries, dead letters, drift detection, and operator approvals repair multi region disaster recovery solution resources without losing the original command or provider evidence.

Deep dives and trade-offs

• Desired versus observed stateKeep multi region disaster recovery solution desired state separate from provider-observed state and show both to operators. Make every reconciliation step conditional and resumable so a worker crash does not restart unsafe effects. Version policy and resource state so old operations cannot overwrite newer intent.
• Provider failures and unknown outcomesUse provider-specific idempotency tokens and query-after-timeout behavior for multi region disaster recovery solution operations. Bound retries with exponential backoff, circuit breakers, and per-provider quotas. Route irreconcilable drift to an approval or quarantine path instead of retrying forever.
• Blast radius and operationsPartition multi region disaster recovery solution work by tenant, region, cluster, or resource class and cap concurrent mutations. Audit who changed desired state, which policy allowed it, and what provider evidence was observed. Alert on drift age, operation backlog, failed steps, policy denials, and stale observations.
• Push versus pull reconciliationUse event triggers for fast response and periodic scans for missed events, drift, and recovery. A push-only multi region disaster recovery solution controller silently misses changes when a provider event is lost.
• Central control plane versus provider-native controllersKeep policy, intent, and audit centralized while isolating provider-specific application logic behind adapters. A monolithic controller becomes hard to scale and couples unrelated provider failure domains.
• Automatic repair versus approvalAutomate low-risk, reversible multi region disaster recovery solution changes and require approval for destructive or high-blast-radius operations. Full automation without policy or blast-radius controls can turn a transient signal into a widespread outage.