Day 2 — Latency, Throughput, and "Numbers Every Engineer Should Know"

Goal: Speak the language of scale fluently. By the end of this lesson you should be able to: - Precisely define latency and throughput, and explain when each matters - Recite (or quickly estimate) the Jeff Dean numbers and use them in back-of-envelope math - Reason about latency percentiles (P50, P99) and the tail latency amplification effect - Apply Little's Law to size a system - Do a back-of-envelope estimate for QPS / storage / bandwidth of a real system - Recognize the latency vs throughput and latency vs consistency trade-offs

Estimated time: 50–60 min read + reflection.

1. Latency vs Throughput — precise definitions

These two words get confused constantly. Get them right in Day 2 and you'll never lose points on them.

The classic analogy: pipes and highways

• Latency = how long a single car takes to drive from A to B.
• Throughput = how many cars per hour arrive at B.

Adding lanes (more workers, more servers) increases throughput without necessarily reducing latency. Making the road shorter (moving closer, better algorithm) reduces latency.

Key insight: you can improve throughput independently of latency by adding parallelism. But you can only reduce latency by making the individual operation faster (better algorithm, closer server, faster hardware, fewer round trips).

When each matters

• Latency dominates for interactive UX — humans notice > 100 ms of lag. A search that takes 2 s at 10 QPS feels awful. Latency = user experience.
• Throughput dominates for background pipelines — a nightly ETL job that processes 10 TB doesn't care if each row takes 50 ms as long as the total throughput hits the SLA.
• Latency and throughput are linked by queueing: high load → queues form → each request waits longer → latency rises. That's why you always size for peak throughput to keep latency in check.

2. Little's Law — the one formula that matters most

For any stable system:

[ L = \lambda \times W ]

Where: - L = average number of items in the system (concurrency, in-flight requests, queue depth) - λ (lambda) = arrival rate (requests per second) - W = average time each item spends in the system (latency, seconds per request)

Practical uses

Sizing a thread pool. Say each request takes 200 ms and you get 500 req/s: - ( L = 500 \times 0.2 = 100 ) → you need at least 100 concurrent workers (threads / goroutines) to keep up.

Explaining why latency explodes at capacity. If you have 100 threads and get 600 req/s, at 200 ms latency you'd need 120 workers. You don't have them, so requests queue → ( W ) grows → latency spikes. This is the "hockey stick" curve on latency graphs.

Setting timeout budgets. If your latency budget is 100 ms end-to-end and each downstream call is 20 ms sequentially, you can afford 5 hops. If you can parallelize, you can afford more.

Memorize Little's Law — it comes up in almost every capacity planning discussion.

3. Numbers every engineer should know (Jeff Dean)

These are the most important numbers in system design. FAANG interviewers love to ask "how long does X take?" — knowing these gives you a 10× intuition head start.

Corollaries you should be able to derive fast

• RAM is ~100× faster than SSD, SSD is ~10× faster than HDD.
• A round trip in the same datacenter (500 µs) is ~2× the time to read 1 MB from RAM. So an in-DC RPC is roughly comparable to a large memory read.
• A cross-continent RTT (150 ms) is 300 000× slower than an L1 cache read. This is why edge / CDN matters so much.
• You cannot beat the speed of light. ~5 ms per 1000 km, one-way, in fiber. Sydney → Virginia (~16 000 km) → ~80 ms one-way, ~160 ms round trip minimum, no matter how much money you throw at it.

How to memorize

Don't try to memorize the whole table cold. Anchor these five key numbers and derive the rest:

1. L1 cache: 1 ns (round up from 0.5)
2. Memory access: 100 ns
3. SSD read (1 MB): 1 ms
4. Same-DC round trip: 0.5 ms
5. Cross-continent RTT: 150 ms

Then remember the rough orders of magnitude between them (1000×, 10×, 300 000×).

4. Percentiles: P50, P95, P99 — why average lies

Averages hide the truth. The average request latency might be 100 ms, but if 1% of requests take 5 s, that's catastrophic if your app has a lot of users.

Definitions

• P50 (median) — half of requests are faster than this.
• P95 — 95% of requests are faster; 5% are slower.
• P99 — 99% of requests are faster; 1% are slower. This is the tail.
• P99.9, P99.99 — even further into the tail. Netflix/Google care about these.

Why the tail matters — the fan-out effect

Modern requests fan out. A single user request might hit 100 backend services in parallel (feed, ads, notifications, storage lookups, ranking, etc.). The user waits for the slowest of the 100.

If each backend has P99 = 100 ms, what's the user-experienced P50?

Probability all 100 responses come back fast = ((0.99)^{100} \approx 0.366) So ~63% of users hit at least one slow response. The user-experienced P50 ≈ backend P99.

Tail latency amplifies at fan-out. This is why FAANG obsesses over P99 (and P99.9) latencies — the tail is the user experience.

Real numbers you should quote

• Google: internal RPCs should have P99 < 10 ms.
• Amazon: an extra 100 ms of page latency = 1% loss in sales.
• User-perceived interactivity: < 100 ms = instant, < 1 s = acceptable, > 3 s = users leave.

5. Bandwidth vs Latency (Stuart Cheshire's classic)

"Bandwidth is not latency." You can buy more bandwidth easily; you cannot buy less latency.

• A 10 Gbps link can send a lot of data per second (bandwidth).
• But a single small packet still takes the same round-trip time (latency).
• Analogy: a wide highway (many lanes) versus a shorter highway. A wider highway doesn't make one car arrive faster.

Implications: - Batching helps throughput/bandwidth utilization, not per-request latency. - Compression trades CPU for bandwidth reduction — good if bandwidth-bound, bad if latency-critical (compression itself takes time). - For chatty protocols (many small requests), latency dominates even on a fast network. Fix by: - Reducing round trips (HTTP/2 multiplexing, batching, GraphQL) - Moving closer (CDN, edge, regional deployment) - Precomputing / caching results

6. Back-of-envelope estimation — the FAANG staple

Every FAANG design interview asks you to estimate scale. Interviewers judge your reasoning, not your accuracy. Round aggressively, show your work.

Useful constants (memorize)

Worked example: Twitter-scale QPS

Given: Twitter has ~300M DAU (Daily Active Users), average user posts 2 tweets/day, and reads (views timelines) 200 times/day.

Writes: - Tweets/day = 300M × 2 = 600M - Tweets/sec (avg) = 600M / 86 400 ≈ 7 000 writes/sec - Peak (typically 4×–10× avg) ≈ 30 000–70 000 writes/sec

Reads: - Timeline reads/day = 300M × 200 = 60B - Timeline reads/sec avg ≈ 700 000/sec - Peak ≈ 3M reads/sec

Read-to-write ratio ≈ 100:1 → design must optimize reads heavily (caching, CDN, fan-out on write).

Worked example: Storage for a URL shortener

Given: 100M new URLs/month, average URL is 100 bytes long, short code is 7 chars.

• New rows/month = 100M
• Bytes per row (short_code + long_url + timestamp + counters) ≈ ~500 bytes with overhead
• Storage per month = 100M × 500B = 50 GB/month
• Storage per year ≈ 600 GB
• With 3× replication → ~2 TB/year

That's small — a single Postgres instance can handle years of this. The interesting design pressure is QPS and latency, not storage.

Worked example: Bandwidth for video streaming

Given: A user streams 1080p video at 5 Mbps for 1 hour.

• 5 Mbps × 3600 s = 18 000 megabits = 2 250 megabytes = ~2.25 GB per hour per user
• 1M concurrent viewers × 5 Mbps = 5 Tbps aggregate egress bandwidth 😱
• This is why you need a CDN — no single origin can push 5 Tbps.

7. Latency vs Throughput trade-off (batching)

Batching groups multiple requests into one, reducing per-request overhead. It improves throughput but adds latency (individual requests wait for the batch to fill up or timeout).

Examples: - Kafka producer batches messages for a few ms → higher throughput, ~5 ms extra latency. - Database bulk insert: 1000 rows per statement is 100× faster than 1000 statements, but the first row waits for the 1000th. - GPU inference: batching many requests together maximizes GPU utilization; sacrifices latency.

Rule of thumb: - Batch when throughput matters more than per-request latency (logs, analytics, ML inference). - Don't batch when latency matters (search, interactive UX, real-time trading).

Related: Latency vs Consistency

• Synchronous replication = write not acknowledged until all replicas ack. High consistency, high latency.
• Asynchronous replication = write acked as soon as primary saves. Low latency, but reads to replicas may be stale.
• We'll go deep on this on Days 11 and 15–16.

8. Where latency comes from in a request (the anatomy)

Ties back to Day 1's 8-step flow. For a single API call from a browser to your service:

Rule of thumb budgets (interactive API): - P50 target: < 100 ms - P99 target: < 500 ms - If you blow either, first ask which step is contributing — don't optimize blindly.

9. Interview angle (what FAANG actually tests)

Common flavors:

• "Estimate the storage and bandwidth for [X system]." — Show your rounding: users → daily events → bytes per event → total per day → per year → × replication factor. Show units at every step.
• "How would you handle 10× more traffic tomorrow?" — Little's Law thinking: is it a throughput problem (add capacity, LB) or a latency problem (cache, precompute, denormalize)?
• "Why does your P99 spike but P50 look fine?" — Talk about GC pauses, cold caches, retries, downstream hiccups, queueing on a hot shard. Show that you know tail latency has multiple causes.
• "Batch these events or process one-by-one?" — Latency vs throughput trade-off; ask what the SLA is.
• "You have a 100 ms latency budget end-to-end and 5 microservices in the call chain — how do you budget?" — Parallel calls fit in one budget slot; sequential calls each eat a slice.

Trap they set: they ask "what's the throughput of a system with X req/s?" — throughput is req/s, so if you say "throughput is X req/s" and stop, you've missed the deeper question ("how much can it handle at target latency?"). Always tie throughput to a latency SLA.

10. Quick reference card (memorize)

• Latency = time per op. Throughput = ops per second. They are not the same thing.
• Little's Law: L = λ × W. Concurrency = arrival rate × time in system.
• P99 > P50 by ~10× in a healthy system, > 100× if the system is sick.
• Fan-out amplifies tail latency: (backend P99) ≈ (user-perceived P50) if fan-out is ~100.
• Bandwidth ≠ latency. You can buy bandwidth; you can't beat physics.
• Batching: better throughput, worse per-request latency.
• Speed of light: ~5 ms per 1000 km one-way in fiber. Sydney ↔ Virginia = ~160 ms RTT minimum.
• Numbers to memorize: L1 = 1 ns · RAM = 100 ns · SSD read (1 MB) = 1 ms · DC RTT = 0.5 ms · cross-continent RTT = 150 ms.

Ready for the Day 2 quiz? Open it in the sidebar (or quizzes/day-02.html). Answer in chat when done.