Day 5 — Caching Fundamentals

Goal: Own the biggest single lever in system design. By the end you should be able to: - Explain why caching wins on latency, throughput, and cost — with the math - Place caches correctly across the 6-layer cache cake (browser → CDN → gateway → app → distributed → DB) - Choose between cache-aside / write-through / write-back / read-through and explain the trade-offs - Pick an eviction policy (LRU / LFU / TTL) with reasoning - Reason about hit rate and its outsized impact on effective latency - Anticipate cache consistency, stale reads, and thundering herd failure modes

Estimated time: 50–60 min read + reflection.

1. Why caching wins — the fundamental economics

Caching answers three problems at once:

• Latency — cached reads are typically 100–10,000× faster than the backing store.
• Throughput — every cache hit is a request that doesn't touch your DB. A 90% hit rate means your DB sees 10% of the traffic — 10× the effective capacity.
• Cost — DB reads are expensive (CPU, IOPS, licenses). Cache reads are cheap RAM ops. Turning DB QPS into cache QPS is one of the highest-leverage cost-savers in engineering.

The effective-latency equation

For a cache-fronted read path:

[ L_{effective} = h \cdot L_{cache} + (1 - h) \cdot L_{miss} ]

Where: - h = hit rate (0 to 1) - L_cache = latency of a cache hit (typical: 1 ms for Redis in-DC) - L_miss = latency of a cache miss (cache + DB + often cache write) — typical: 20–100 ms

Worked example. DB read = 50 ms, cache hit = 1 ms. - At 0% hit rate: effective = 50 ms (cache is dead weight) - At 50% hit rate: effective = 0.5 × 1 + 0.5 × 50 = 25.5 ms (2× faster) - At 90% hit rate: effective = 0.9 × 1 + 0.1 × 50 = 5.9 ms (8× faster) - At 99% hit rate: effective = 0.99 × 1 + 0.01 × 50 = 1.49 ms (33× faster)

Hit rate is not linear in its effect. Small improvements at the top of the curve matter enormously.

2. The 6-layer cache cake

Real systems cache in many places at once. When someone asks "where should we cache?" the answer is usually "yes — all of these":

┌─────────────────────────────────────────────────────┐
│  1. Browser cache          (Cache-Control, ETag)     │
├─────────────────────────────────────────────────────┤
│  2. CDN / edge cache       (CloudFront, Cloudflare)  │
├─────────────────────────────────────────────────────┤
│  3. Reverse proxy cache    (Varnish, NGINX)          │
├─────────────────────────────────────────────────────┤
│  4. Application cache      (in-process: Caffeine)    │
├─────────────────────────────────────────────────────┤
│  5. Distributed cache      (Redis, Memcached)        │
├─────────────────────────────────────────────────────┤
│  6. Database's own cache   (buffer pool, plan cache) │
└─────────────────────────────────────────────────────┘
                       ↓
                     Storage

Layer-by-layer

1. Browser cache — client keeps assets in local memory/disk based on HTTP headers (Cache-Control: max-age=3600, ETag, Last-Modified). - What to cache: static assets (CSS, JS, images, fonts), sometimes JSON responses. - Wins: 0 ms latency, 0 server load, 0 bandwidth cost. - Traps: stale content when you deploy; use cache-busting URL hashes (app.a1b2c3.js).

2. CDN / edge cache — a network of geographically distributed servers holding cached content close to users. - What to cache: anything cacheable (static + some API responses via Cache-Control: s-maxage=X). - Wins: cross-continent latency drops from ~150 ms to ~20 ms (users hit their nearest edge). - Traps: invalidation is slow (purge API takes seconds), you don't see cache misses in your metrics unless you look for them. - Real cost saver: Netflix serves 95%+ of video bytes from CDN edges (Open Connect), not origin.

3. Reverse proxy cache — same-DC HTTP cache in front of your app (Varnish, NGINX, Envoy). - What to cache: rendered HTML pages, API responses that vary by URL/headers. - Wins: takes CPU-heavy render work off your app servers. - Traps: cache-key design is tricky (should you vary by cookie? user? locale?).

4. Application-level cache (in-process) — cache inside the app process (Java Caffeine, Node lru-cache, Python functools.lru_cache). - What to cache: hot lookups that don't need to be shared across servers (config, feature flags, hot user profiles). - Wins: microsecond latency, no network hop. - Traps: inconsistent across servers (each app has its own copy) — cache invalidation across N app servers is hard.

5. Distributed cache — Redis, Memcached, or a managed service (ElastiCache, DynamoDB DAX). - What to cache: anything shared across app servers — session state, hot DB queries, rate-limit counters, computed values. - Wins: microservices architecture's default cache layer. One source of truth for cached data across the fleet. - Traps: network hop (~1 ms in-DC), you own capacity + eviction + fault tolerance.

6. Database's own cache — the DB itself has a buffer pool (Postgres shared_buffers, MySQL InnoDB buffer pool) and query plan cache. - What to cache: the DB does this for you — recently-read pages, prepared statement plans. - Wins: essentially free; well-tuned DBs serve 95%+ of reads from RAM. - Traps: if your working set > buffer pool → thrashing, disk IO explodes. Right-size your DB instance.

Design principle: every layer you skip is 10× the latency of the layer above. Cache as high in the stack as possible — a hit at the browser is free; a hit at Redis costs 1 ms + a network round-trip; a hit at Postgres buffer pool costs 100–500 µs + a network round-trip + query parsing.

3. Cache write strategies — how does the cache and DB stay in sync?

Four canonical patterns. Know all four and their trade-offs.

3.1 Cache-aside (a.k.a. lazy loading) — most common

The app manages the cache explicitly.

Read path:
  1. Look in cache. Hit? Return.
  2. Miss? Read from DB.
  3. Populate cache.
  4. Return.

Write path:
  1. Write to DB.
  2. Invalidate or update cache.

• Pros: simple, works with any cache, only cached data is what's been accessed (no cold cache pollution).
• Cons: every miss = one extra RTT; cache and DB can diverge if writes fail partway.
• Use for: most read-heavy workloads. Redis + Postgres + your app doing it manually.

3.2 Read-through

The cache itself knows how to load from the DB on miss (encapsulated inside the cache client or a library).

Read path:
  1. App asks cache.
  2. Cache hit? Return.
  3. Miss? Cache internally reads from DB and populates itself.
  4. Return.

• Pros: app code is simpler (just talks to cache).
• Cons: cache library must know how to talk to your DB; less flexibility.
• Use for: systems with a cache-abstraction library (e.g. AWS DAX in front of DynamoDB).

3.3 Write-through

Writes go synchronously through the cache to the DB.

Write path:
  1. App writes to cache.
  2. Cache writes to DB (blocking).
  3. Ack back to app.

• Pros: cache is always consistent with DB.
• Cons: writes are slower (must wait for DB); cold cache pollution (every write goes in even if you never read it).
• Use for: write-heavy workloads where cache-consistency matters.

3.4 Write-back (write-behind)

Writes go to cache; cache flushes to DB asynchronously.

Write path:
  1. App writes to cache. Ack immediately.
  2. Cache flushes to DB later (batched).

• Pros: blazing-fast writes; batching amortizes DB overhead.
• Cons: you can lose writes on cache crash (data was acknowledged but never persisted). Requires the cache to be durable enough to survive.
• Use for: high-write workloads where some data loss is acceptable (analytics counters, metrics) or where the cache itself is durable (Kafka + downstream flusher).

3.5 Comparison at a glance

4. Cache eviction policies

Cache is finite. When it's full, something gets kicked out. Which one?

4.1 LRU (Least Recently Used) — default and safe

Evict whatever was accessed the longest time ago.

• Great for: temporal locality (recent things get accessed again). Almost all workloads have this.
• Bad for: scan-heavy workloads that touch every item once (destroys the cache).
• Standard in Redis with maxmemory-policy allkeys-lru.

4.2 LFU (Least Frequently Used)

Evict whatever has the lowest access count.

• Great for: long-term popularity (a Wikipedia page that's been accessed 1M times shouldn't get evicted just because a burst of new pages came in).
• Bad for: trends that change (yesterday's hot page stays cached even after everyone's moved on).
• Fixed by: LFU with aging or LFU-DA (dynamic aging) or W-TinyLFU (used by Caffeine).

4.3 TTL-based (Time To Live)

Every entry has an expiration; when time's up, it's evicted regardless of access.

• Great for: data with natural freshness bounds (session expires in 30 min; stock prices expire in 5 s).
• Combine with LRU/LFU: most caches use TTL plus LRU — evict expired first, then LRU-tail if still full.

4.4 FIFO / random

Rarely optimal but sometimes fine (random is surprisingly OK for high-throughput caches — no bookkeeping, and cache lines all become "one of many").

4.5 Practical decision

• Start with LRU + TTL. Covers 90% of cases.
• Move to LFU/W-TinyLFU for content-recommendation-style workloads with long-tail popularity.
• Switch to write-through with no eviction for small, hot lookup tables (feature flags).

5. Cache consistency — "the second hard thing"

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton

Why cache invalidation is hard: 1. Stale data — an update happens; cache still serves old value. How long before all clients see the new one? 2. Race conditions — concurrent write + read can leave cache and DB inconsistent forever (writer sets cache to new value; reader misses, reads old value from DB replica lag, overwrites cache with old value). 3. Distributed invalidation — with N app servers and M cache nodes, telling everyone is a coordination problem.

Common patterns to reason about

TTL-based freshness. Set a TTL that matches your consistency budget. Users can tolerate 30-second-stale product prices? TTL = 30 s. Simple, cheap, "eventually consistent."

Cache invalidation on write (cache-aside):

1. Write DB.
2. DELETE from cache.
3. Next read: cache miss → repopulate.

Simpler than trying to update the cache — deletion is idempotent and race-safe.

Versioned keys / cache stampede avoidance: - Use versioned keys: user:42:v7 — bump the version on any change; old entries expire on their own. - Use request coalescing (a.k.a. single-flight): when 1000 threads all miss the same key, only ONE of them queries the DB; the rest wait and use the result. Prevents the thundering herd.

Read-your-own-writes. After a user updates their profile, they immediately reload — and see the old value. Confusing. Fixes: - Write to cache and DB (double-write). - For that user's next N reads, force a DB read (session-scoped consistency). - Route the user's traffic to the primary (session pinning).

Cache stampede / thundering herd

When a popular key expires and 10,000 concurrent readers all miss simultaneously → 10,000 DB queries → DB melts.

Mitigations: - Request coalescing / single-flight. (Go's singleflight package, Java's AsyncLoadingCache.) - Probabilistic early expiration. Some readers refresh the cache before it expires (e.g. XFetch algorithm). - Locking on cache miss. First reader takes a lock, populates cache, releases; others wait. - Serve stale + refresh in background. Return the stale value immediately, queue a background refresh. (stale-while-revalidate header.)

6. Hot keys — the shard-buster

A "hot key" is a cache key that receives disproportionate traffic (e.g. a celebrity's profile on a social app).

• One Redis shard handles that key → that shard saturates while others idle.
• Classic P99 spike source: request queueing on a single shard.

Mitigations: - Replication of hot keys across multiple shards (client picks randomly). - Client-side caching for the hottest keys (a level above distributed cache). - Sharding by (key, sub-key) to split the load. - Detect them: log which keys make up top 1% of traffic.

7. Interview angle (what FAANG asks)

Common flavors:

• "How would you speed up this slow endpoint?" → Walk the 6-layer cake. Ask about read/write ratio, staleness tolerance, key cardinality.
• "Cache-aside vs write-through vs write-back — which and why?" → Section 3.5 matrix. Match to consistency and read/write pattern.
• "What's the effective latency with a 95% hit rate?" → Use the equation. Show your work.
• "How do you invalidate the cache when the DB changes?" → TTL + delete-on-write for most cases. For strict consistency, DB CDC (change-data-capture) via Kafka → cache invalidation service.
• "How would you handle a cache stampede?" → Single-flight, probabilistic early expiration, stale-while-revalidate.
• "You've got hot keys — what do you do?" → Section 6.

Trap they set: they'll say "just add a cache and we're done." A weak candidate says "yes." A strong candidate asks: - What's the read/write ratio? (Cache wins on read-heavy workloads.) - What's the staleness tolerance? (Drives TTL, invalidation strategy.) - What's the key cardinality? (Too many unique keys = low hit rate.) - What's the hit rate we can expect? (Which drives whether cache is worth its cost.)

8. Quick reference card

• Cache wins: latency ↓, throughput ↑, cost ↓. Hit rate has non-linear effect on effective latency.
• 6-layer cake: Browser → CDN → Reverse proxy → App-local → Distributed (Redis) → DB buffer pool.
• Write strategies: Cache-aside (default), Read-through, Write-through (consistent, slow), Write-back (fast, risky).
• Eviction: LRU + TTL (default); LFU/W-TinyLFU for long-tail popularity.
• Consistency: TTL for eventual; delete-on-write for cache-aside; versioned keys for correctness under races.
• Stampede prevention: single-flight, probabilistic early expiration, stale-while-revalidate.
• Hot keys: replicate across shards, client-side cache, sub-key sharding.
• Cache invalidation is one of the two truly hard things — always be explicit about TTLs and staleness bounds.

Ready for the Day 5 quiz? Open it in the sidebar. Includes a spaced-repetition callback on tail-latency amplification (Day 2 Q3 — the "recognize when to reach for the formula" skill). Answer in chat when done.