Day 4 — Load Balancers Deep Dive

Goal: Own the load balancer — the box that sits at the top of nearly every design diagram. By the end you should be able to: - Distinguish L4 vs L7 load balancers precisely (beyond the diagnostic-level answer) - Choose the right load-balancing algorithm for a workload - Design health checks that catch real failures without flapping - Reason about sticky sessions and when they hurt more than help - Design high availability of the LB itself (LBs are SPOFs if you're not careful) - Walk through multi-region LB architecture end-to-end

Estimated time: 50–60 min read + reflection.

1. Recap — what a load balancer actually does

A load balancer sits between clients and a pool of servers ("backends") and distributes incoming requests across them. It does three jobs:

1. Distribute load — spread traffic so no single backend melts.
2. Health-check backends — stop sending traffic to broken ones.
3. Provide a single stable address — clients hit api.example.com regardless of how many backends exist behind it (this decouples the topology from the client).

Everything else — SSL termination, rate limiting, caching, WAF, connection pooling — is a feature built on top of these three jobs.

2. Layer 4 vs Layer 7 — deeper

You got the surface-level answer on the diagnostic. Now the depth.

Layer 4 (transport-layer)

• Operates on TCP/UDP — sees IPs, ports, and the raw byte stream.
• Does not inspect payload (doesn't parse HTTP, doesn't see URLs or headers).
• Forwards packets, sometimes just re-writing the destination IP (DSR, NAT modes).
• Fast — no parsing, kernel-level or ASIC-accelerated, million-plus PPS per box possible.
• Preserves the entire connection to one backend — the same client's whole TCP session goes to the same backend.

When to use L4: - Non-HTTP traffic (databases, MQTT, custom TCP protocols, WebSocket connections that need raw pass-through). - Ultra-low-latency, ultra-high-throughput (financial market data, gaming lobbies). - When you don't want the LB to terminate TLS (end-to-end encryption pass-through).

Examples: AWS NLB, GCP TCP/SSL Proxy, HAProxy in TCP mode, IPVS.

Layer 7 (application-layer)

• Operates on HTTP/HTTPS/gRPC — parses the request, sees the URL, headers, cookies, method, body.
• Can route on content: /api/* → API pool; /static/* → static pool; Host: v2.example.com → v2 backends.
• Can rewrite requests (add headers, strip paths, canary a percentage of traffic).
• Terminates TLS by default (so it can decrypt and read the request).
• Slower than L4 (extra CPU per request), but per-request features are worth it for HTTP.

When to use L7: - Almost all HTTP APIs and web traffic — the default. - When you need content-based routing (path, header, cookie). - When you need per-endpoint rate limits, canary deploys, A/B routing. - When you want the LB to handle auth, rate limiting, JWT validation, or WAF rules.

Examples: AWS ALB, NGINX, HAProxy in HTTP mode, Envoy, Traefik, GCP HTTP(S) LB, Cloudflare.

The nuance FAANG looks for

• L7 can do more, but costs more CPU and adds latency (~1–5 ms).
• L4 sees raw connections; L7 sees requests. A single L4 connection can carry many L7 requests (HTTP/2 multiplexing).
• You often stack them: L4 in front (for raw throughput and TLS pass-through) → L7 behind (for HTTP routing per-service). AWS pattern: NLB → ALB → app.

3. Load-balancing algorithms

Picking the algorithm is a real interview question — don't guess, know the trade-offs.

3.1 Round Robin

Cycle through backends in order: A, B, C, A, B, C…

• Pros: dead simple, no state.
• Cons: ignores backend load. If one request happens to be huge (long-running query), one backend gets stuck while others idle.
• Best for: homogeneous backends serving homogeneous, short requests.

3.2 Weighted Round Robin

Each backend has a weight. A backend with weight 3 gets 3× the traffic.

• Use when backends have different capacities (e.g. new bigger nodes added to fleet).
• Also used in gradual rollouts (canary: 5% traffic to the new version).

3.3 Least Connections

Send the next request to the backend with the fewest active connections.

• Pros: adapts to real-time load. A backend stuck on a long request stops receiving new work.
• Cons: needs the LB to track connection state — slightly more expensive.
• Best for: long-lived connections (WebSockets, DB proxies) or heterogeneous request durations.

3.4 Least Response Time

Route based on backend response latency (moving average).

• Sends more traffic to faster backends. Good when backends are heterogeneous or one is degraded.
• Downside: unstable in bursty conditions (a fast backend can suddenly get all traffic and swamp itself).

3.5 IP Hash / Consistent Hash

Compute hash(client_ip) (or hash(session_id) or hash(user_id)) and mod it into the backend list. Same key → same backend.

• Use case: cache locality (the same user hits the same cache-warm backend), sticky sessions, sharded systems.
• Naive hash breaks when a backend is added/removed (all keys re-map).
• Consistent hashing (Day 13) minimizes remapping — only ~1/N keys move. We'll build this out fully next week.

3.6 Power of Two Random Choices (P2C) — the surprising winner

Pick 2 backends at random; send the request to the one with fewer active connections.

• Basically as good as full "least connections" but with a fraction of the coordination overhead.
• Standard in modern systems: Envoy, Finagle, gRPC LBs default to P2C.
• Interview-quality fact to drop.

3.7 Quick algorithm decision matrix

4. Health checks — the hard part

A load balancer's promise is "only send traffic to healthy backends." Getting that right is subtle.

4.1 Active vs Passive health checks

• Active: LB actively calls a health endpoint (GET /healthz) every N seconds. If it fails M times in a row → mark backend unhealthy.
• Passive: LB observes real request failures. If a backend fails K requests in a window → mark unhealthy.

Most LBs do both.

4.2 What should /healthz actually check?

Three flavors — pick deliberately:

The classic mistake: making /healthz check every downstream (DB + cache + queue + …). If one downstream is briefly slow, all backends fail health check simultaneously → LB removes them all → cascading outage from a soft downstream hiccup. Rule of thumb: check what this instance needs to serve traffic, not the health of the whole system.

4.3 Flapping

If health thresholds are too aggressive, a backend can flap in and out of rotation. Standard mitigations:

• Consecutive checks: require N consecutive failures to remove; M consecutive successes to add back.
• Hysteresis / cooldown: don't re-add for at least K seconds after removal.
• Rate limiting the removal decisions across the fleet (don't remove more than X% of the fleet in Y seconds).

4.4 Slow-start problem

When you add a fresh backend, don't immediately send it 1/N of traffic. Its caches are cold, JIT hasn't warmed, connection pools aren't warm. It'll be slow → its response-time-based algorithms will route away from it → but round-robin will hammer it.

Fix: connection ramp-up (slow-start) — the LB gives new backends a fraction of traffic that ramps to full over ~30–60 seconds. AWS ALB, Envoy, NGINX all support this.

5. Sticky sessions (session affinity)

"Route this client to the same backend every time."

When you need it

• WebSockets — the connection is inherently pinned to one backend for its lifetime (Day 3).
• Legacy apps that store session state in-process — if you can't fix the app to externalize state to Redis, sticky sessions are the workaround.
• Cache locality wins — Consistent hashing by user ID keeps a user's hot data on the same instance.

When it hurts

• Uneven load — heavy users pin to specific backends; those backends melt.
• Scaling is painful — adding a backend doesn't relieve pressure on already-pinned clients.
• Failure blast radius — when a backend dies, all its pinned clients lose sessions.
• Deploys are messy — rolling restart forces mass re-pinning.

How it's implemented

• Cookie-based: LB sets a cookie like AWSALB=backend-id; client echoes it, LB reads it, routes accordingly. Works through NATs.
• IP-based: hash the client IP. Breaks behind CGNAT/mobile carriers (many users share one IP).
• Header/token-based: custom app header carries a routing hint.

The FAANG answer

"Prefer stateless services + externalized session state (Redis) over sticky sessions. Reach for stickiness only when the workload is inherently stateful (WebSockets, in-memory session, cache-warmup optimization)."

6. TLS termination

Where do you decrypt HTTPS?

6.1 At the LB (most common)

Client ──HTTPS──▶ LB ──HTTP (plaintext) or HTTPS (re-encrypted)──▶ backends

• Pros: LB handles cert management, cipher upgrades, TLS 1.3, session resumption; backends see cheap HTTP; L7 features (routing, WAF) can inspect the request.
• Cons: LB has the private key (secure it); traffic between LB and backend is plaintext (fine inside a VPC, encrypt anyway for defense in depth).

6.2 TLS passthrough

Client ──HTTPS──▶ L4 LB ──HTTPS untouched──▶ backends

• Pros: end-to-end encryption; backends see real client cert (mTLS).
• Cons: LB can't do L7 features; per-backend certificate management is a chore.

6.3 Re-encryption

Client ──HTTPS──▶ LB (terminates) ──new HTTPS to backend──▶ backends

• Adds latency (double handshake) but gives full L7 features and end-to-end encryption.
• Common for regulated workloads.

7. High availability of the LB itself

A single LB is a single point of failure. Real designs use these patterns:

7.1 Active-passive (VRRP / floating IP)

Two LBs; one active, one hot-standby. If the active dies, the passive takes over the shared IP (via VRRP or a cloud-managed floating IP). Failover in 1–5 seconds typically.

7.2 Active-active (multiple LBs, DNS or Anycast)

• DNS round-robin returns multiple LB IPs → clients pick one. Simple, but DNS caching means dead LBs stay in rotation until TTL expires (Day 1 problem!).
• Anycast — the same IP is announced by multiple LB clusters in different regions/DCs. BGP routes each client to the nearest healthy cluster. Failure is handled at the network layer in seconds.

7.3 Managed cloud LBs

AWS ALB/NLB, GCP LB, Cloudflare — under the hood, cloud providers run these as huge distributed systems with built-in HA. You get a stable DNS name; they handle the rest. The right answer for 95% of designs.

8. Multi-tier LB architecture (the real pattern)

Real large-scale systems don't have "a load balancer" — they have layers of them:

                       ┌───── Global Traffic Manager (Anycast / DNS geo-routing)
                       │      route users to their nearest region
                       ▼
                  ┌────────┐
                  │ Region │
                  │   LB   │  ← L4/L7, terminates TLS, routes to zones
                  └────────┘
                       │
              ┌────────┼────────┐
              ▼        ▼        ▼
          AZ-1 LB   AZ-2 LB   AZ-3 LB    ← per-availability-zone LB
              │        │        │
           app-1    app-2    app-3     ← app tier
              │
              ▼
       ┌────────────┐
       │ Service LB │  ← service-to-service (Envoy sidecar, service mesh)
       └────────────┘
              │
              ▼
        DB Proxy (PgBouncer)   ← connection-pool "LB" in front of DB

Each tier serves a specific purpose:

9. Common LB failure modes

9.1 Cascading failure from bad health check

Symptom: one flaky downstream causes all backends to fail /healthz → LB removes them all → outage. Fix: don't check downstream health in /healthz; use your readiness only.

9.2 Thundering herd on backend restart

Symptom: after a backend restarts, LB adds it back and slams it with 1/N of traffic → it crashes cold caches. Fix: slow-start ramp-up (Section 4.4).

9.3 Hot backend from sticky sessions

Symptom: one user with heavy traffic pinned to backend-3 → backend-3 saturates; others idle. Fix: consistent hashing with bounded loads; remove stickiness for that endpoint.

9.4 Connection storm on LB failover

Symptom: LB-A fails → all clients reconnect to LB-B simultaneously → LB-B melts. Fix: client-side exponential backoff + jitter (Day 3 Q9).

9.5 Retry amplification through the LB tiers

Symptom: 1 failed request → app retries 3× → LB retries each 3× → 9× amplification at the DB. Fix: retries at one layer only; circuit breakers; retry budgets (Netflix Concurrency Limits).

10. Interview angle (what FAANG actually asks)

Common flavors: - "Walk me through where you'd place LBs in this design." → Multi-tier answer from Section 8. Show you know that "load balancer" isn't one box. - "L4 or L7 for [X]?" → Ask about protocol first (HTTP → L7 default, non-HTTP → L4). Then feature needs (content routing → must be L7). - "Which algorithm and why?" → Section 3.7 matrix. Show that you know P2C exists (differentiator). - "How do you health-check a service that depends on a database?" → Don't include DB health in your /healthz; that couples failure modes. Use a per-instance readiness signal. - "How does the load balancer stay highly available itself?" → Active-passive VRRP, active-active Anycast, or managed cloud LB. Discuss failure detection speed and blast radius. - "What are sticky sessions and when would you avoid them?" → Section 5. Always prefer stateless + externalized state.

Trap they set: they'll say "the LB is showing a hot backend, the others are cold — what's happening?" A weak candidate reaches for algorithm choice. A strong candidate asks: is stickiness on? Is there consistent hashing on a low-cardinality key? Is there slow-start delaying the hot backend from shedding load to fresh instances?

11. Quick reference card

• LB does 3 things: distribute load, health-check backends, provide a stable address.
• L4 = fast, connection-level (TCP/UDP, no HTTP parsing). L7 = smart, request-level (HTTP/HTTPS, content-based routing).
• Algorithms: Round Robin (default), Least Connections (long-lived), Consistent Hash (cache locality), P2C (modern default).
• Health checks: keep /healthz local — don't check downstream health, or you cascade.
• Sticky sessions: avoid unless the workload is inherently stateful (WebSockets); externalize state to Redis instead.
• TLS termination: at LB most commonly (backends see plain HTTP inside VPC).
• HA for the LB itself: Anycast (best), active-passive VRRP, active-active DNS, or managed cloud LB.
• Multi-tier LB is normal: global → regional → zonal → service mesh → DB proxy.
• Beware retry amplification through multiple LB layers.

Ready for the Day 4 quiz? Open it in the sidebar (quizzes/day-04.html). Includes two spaced-repetition callbacks (JWT vs sessions from Day 1; bandwidth vs latency from Day 2). Answer in chat when done.