Latency Numbers

System Design FundamentalsMidsystem-design

The Question

What are the key latency numbers every engineer should know?

What a Strong Answer Covers

• Four numbers in correct order of magnitude.

Senior-Level Answer

Jeff Dean's latency numbers (periodically updated as hardware improves) are the calibration tool for system design. The orders of magnitude matter more than the exact figures.

**The canonical table:**

| Operation | Latency | Notes | |---|---|---| | L1 cache reference | ~1 ns | | | L2 cache reference | ~4 ns | 4× L1 | | L3 cache reference | ~40 ns | 10× L2 | | Main memory (DRAM) | ~100 ns | 100× L1 | | NVMe SSD read | ~100 µs | 1,000× memory | | HDD seek | ~10 ms | 100× SSD | | Read 1 MB sequentially from memory | ~25 µs | | | Read 1 MB sequentially from SSD | ~1 ms | | | Round-trip in same datacenter | ~500 µs | | | Cross-region network (US-EU) | ~150 ms | 300× datacenter RTT | | DNS lookup | ~20–100 ms | Varies |

**The ratios that matter most for architecture:**

- Memory is ~1,000× faster than SSD — keep hot data in RAM - SSD is ~100× faster than HDD — use SSD for latency-sensitive workloads - In-datacenter network is ~300× faster than cross-region — minimize cross-region synchronous calls - CPU operations are ~100,000× faster than disk — a tight loop hitting disk on every iteration is catastrophic

**Derived implications:** - A cache miss that hits the database costs ~500 µs (network) + query time; Redis keeps this under 1 ms total - Synchronous cross-region calls in the critical path add 150 ms+ — use async replication or CDN for user-facing latency - Sequential disk reads are 10–100× faster than random — batch and sort writes (LSM trees, log-structured storage)

These numbers change as hardware evolves (NVMe is 10× faster than SATA SSD) but the relative order and ratios are stable benchmarks.

What Separates a 2/3 from a 3/3

Common Mistakes

• Getting memory and SSD latency in the same ballpark — they are 1,000× apart
• Forgetting that in-datacenter RTT is ~0.5ms — not 'near zero'
• Not knowing the numbers but also not knowing the ratios — the ratios are more useful than the exact figures
• Confusing throughput (MB/s) with latency (ms) — a system can have high throughput and poor latency

Follow-Up Questions

• Why is Redis so effective as a cache given these latency numbers? — Redis serves from memory (~1µs) vs a database round-trip (~1ms+) — 1,000× latency reduction for cache hits.
• What do these numbers imply about synchronous cross-region microservice calls? — 150ms+ per hop — a chain of 5 synchronous cross-region calls adds 750ms to user-facing latency; use async.
• How do LSM trees exploit the sequential vs random IO latency difference? — LSM trees batch writes into sequential log appends (fast) and compact offline; avoids random writes that cost 10× more.
• How would you use latency numbers to argue for or against a caching layer? — If uncached response = 1ms DB query + 500µs network and cache hit = 10µs, the math justifies cache for > 10K QPS.

Related Questions

• Redis Caching Patterns
• Vertical vs Horizontal Scaling
• API Versioning
• SLOs vs SLAs
• Availability — Five 9s