Load Balancer Failover

Distributed SystemsSeniorload-balancing

The Question

How does load balancer failover work?

What a Strong Answer Covers

• Active/passive
• heartbeat
• health checks.

Senior-Level Answer

Load balancer failover is the process by which a load balancer detects that a backend instance has become unhealthy and automatically removes it from the pool of eligible targets, redirecting traffic to healthy instances without manual intervention.

**Health checks** are the detection mechanism. Active health checks: the load balancer proactively sends periodic probe requests (HTTP GET to `/health`, TCP connect, or ICMP ping) to each backend. If a backend fails to respond within a timeout or returns a non-2xx status for N consecutive checks, it is marked unhealthy and removed from rotation. Passive health checks: the load balancer monitors real request responses—if a backend returns 5xx errors or times out above a threshold, it is marked unhealthy. Passive checks have lower overhead but detect failures only when real traffic is affected.

**Failover sequence**: (1) Health check detects failure. (2) Load balancer marks instance unhealthy, stops routing new requests to it. (3) In-flight requests to the failed instance may be retried (if the method is idempotent and the load balancer is configured for retries) or returned as errors to the client. (4) Traffic redistributes across remaining healthy instances. (5) If all instances fail, the load balancer returns 503 Service Unavailable.

**Session persistence (sticky sessions)** complicates failover: if sessions are pinned to a specific backend via cookie or IP hash, a backend failure breaks all sessions on that node. Mitigation: store session state externally (Redis, database) so any backend can serve any session.

**Connection draining (deregistration delay)**: before removing an instance (for scale-in or deployment), the load balancer stops routing new requests but allows in-flight requests to complete, up to a configurable timeout. This prevents 5xx spikes during rolling deploys.

**Active-passive vs. active-active**: In active-passive, the secondary load balancer is idle until the primary fails (detected via heartbeat/VRRP). In active-active, multiple load balancers share traffic via DNS round-robin or anycast. Active-active has no cold-start failover penalty but requires all nodes to handle full load.

The practical tuning levers are health check interval, consecutive failure threshold, and connection drain timeout—too sensitive causes flapping; too lenient delays failover during real outages.

What Separates a 2/3 from a 3/3

Common Mistakes

• Assuming health checks are always active—not mentioning passive health check mode.
• Not discussing connection draining, which causes dropped requests during rolling deploys.
• Ignoring sticky session complications during failover.
• Not distinguishing load balancer failover (L4/L7) from upstream service failover.

Follow-Up Questions

• How do you prevent health check flapping when a backend is intermittently slow? — Use a consecutive failure threshold (e.g., 3 failures before marking unhealthy) and a recovery threshold (e.g., 2 successes to re-add) with appropriate timeouts.
• What happens to in-flight requests when a backend is removed from rotation? — Depends on config: connection draining lets them complete; without draining, TCP RST or timeout. Idempotent requests can be retried by the LB.
• How does an L4 load balancer failover differ from an L7 load balancer? — L4 operates at TCP/IP—only TCP connect health checks; no HTTP-level awareness. L7 can inspect HTTP responses, headers, and URLs for smarter routing.
• How would you design a zero-downtime deployment using load balancer features? — Rolling deploy: take instances out of rotation one by one (with drain), deploy, health-check passes, re-add. Blue-green: shift 100% traffic atomically at LB layer.

Related Questions

• Consistent Hashing
• Rate Limiting
• CDN
• Load Balancer — L4 vs L7
• Distributed Transactions — 2PC, Saga