Multi-data Center

System Design FundamentalsSeniorsystem-design

The Question

What are the challenges of a multi-data center architecture?

What a Strong Answer Covers

• Geo-DNS
• async replication
• failover.

Senior-Level Answer

Running across multiple data centers (or regions) eliminates single points of geographic failure and reduces latency by serving users from the nearest location. However, it introduces a set of non-trivial challenges:

**Data consistency and replication lag**: writes to one data center must propagate to others. During the replication window, reads from a secondary region may return stale data. Systems must choose a consistency model: strong consistency (synchronous replication — high latency), eventual consistency (async replication — stale reads possible), or causal consistency (partial ordering guarantees).

**Traffic routing and DNS failover**: GeoDNS routes users to the nearest healthy data center. DNS TTLs are typically 60–300 seconds, so failover is not instantaneous. Applications must handle the window between a data center going down and DNS propagating the change — connection pooling, retries, and circuit breakers reduce user impact.

**Failover automation**: detecting a failed data center reliably is harder than it sounds. Split-brain scenarios occur when a data center is isolated (network partition) but not down — two regions both believe they are primary and accept writes. Solving this requires a quorum-based consensus mechanism (Raft, Paxos) or explicit tie-breaking logic.

**Data residency and compliance**: GDPR and similar regulations may prohibit replicating EU user data to US data centers. This creates partitioned data tiers where not all data can be multi-region.

**Deployment complexity**: deploying a new version across regions requires coordination — deploy to one region, validate, then proceed. Schema migrations must be backward-compatible with both old and new application versions simultaneously (expand/contract pattern).

**Cost**: cross-region data transfer is expensive. Minimizing cross-region traffic through caching, read replicas, and careful routing is essential.

The canonical pattern: active-active for stateless web/cache tiers, active-passive or quorum-based for the write path to the primary database. Global load balancing (Route 53 health checks, Cloudflare) handles traffic steering.

What Separates a 2/3 from a 3/3

Common Mistakes

• Treating multi-region as purely a DNS routing problem — ignoring write consistency
• Not mentioning split-brain — a critical failure mode that invalidates naive failover
• Ignoring compliance constraints on data replication
• Describing only the happy path — missing failure modes during the failover window

Follow-Up Questions

• How do you handle a schema migration that must run across two active regions? — Expand/contract: deploy backward-compatible schema change first, then update code, then clean up old columns — never break both regions simultaneously.
• What is split-brain and how do you prevent it? — Both partitions believe they are primary and accept writes — prevent via quorum (need majority to accept writes) or a fencing token.
• How does GeoDNS failover work and what are its limitations? — Route 53 health checks detect failure and update DNS — TTL caching means clients can point to dead endpoints for minutes.
• When would you choose active-passive over active-active for your database tier? — When write conflicts are too complex to resolve — active-passive avoids conflicts at the cost of failover downtime and capacity.

Related Questions

• Redis Caching Patterns
• Vertical vs Horizontal Scaling
• API Versioning
• SLOs vs SLAs
• Availability — Five 9s