Latency vs Throughput

System Design FundamentalsEntrysystem-design

The Question

What is the difference between latency and throughput?

What a Strong Answer Covers

• Both defined
• batching tradeoff.

Senior-Level Answer

Latency and throughput are the two primary axes of performance, and they measure fundamentally different things.

**Latency** is the time elapsed between a request being sent and its response being received — the 'delay' experienced by a single operation. It is typically measured in milliseconds (or microseconds for high-performance systems) and reported at percentiles: p50 (median), p95, p99, p99.9. The tail latency (p99, p99.9) is often more operationally important than the median because it represents the worst-case experience for a real fraction of users. A system can have a p50 latency of 10ms and a p99 latency of 2 seconds, masking severe tail behavior.

**Throughput** is the amount of work a system can complete in a unit of time — typically measured in requests per second (RPS), transactions per second (TPS), or data transfer rate (MB/s). Throughput describes the system's capacity ceiling and is the primary metric for batch processing, data pipelines, and high-volume APIs.

The relationship between latency and throughput is described by **Little's Law**: `L = λW`, where L is the average number of requests in the system, λ is the arrival rate (throughput), and W is the average time in system (latency). As throughput approaches a system's capacity, queuing delays increase latency sharply — this is the fundamental trade-off. At low load, latency is dominated by processing time. At high load, it is dominated by queue wait time.

In practice, they often trade off: batching operations improves throughput by amortizing per-request overhead but increases latency for individual requests. Conversely, parallelism can reduce latency for individual complex queries but may reduce overall throughput by increasing resource contention.

Optimization strategies differ: improving latency requires reducing coordination overhead, co-locating data with computation, and minimizing lock contention. Improving throughput requires maximizing parallelism, minimizing I/O stalls, and efficient use of CPU pipelines and memory bandwidth.

Good system design establishes separate SLOs for latency (p99 < 200ms) and throughput (system handles 10,000 RPS) — they require different optimization approaches and measurement strategies.

What Separates a 2/3 from a 3/3

Common Mistakes

• Using average latency as the primary metric — p99/p99.9 are more operationally relevant.
• Treating latency and throughput as independent when they are coupled through Little's Law.
• Claiming you can always optimize both simultaneously — at high utilization there is an inherent trade-off.
• Confusing throughput with bandwidth — throughput is application-level operations; bandwidth is link-level bits per second.

Follow-Up Questions

• A service has p50 latency of 5ms and p99 latency of 3s. What does this suggest and how would you investigate? — Severe tail latency — suggests GC pauses, lock contention, slow external calls, or hot partitions. Investigate with percentile flame graphs, distributed tracing, and slow query logs.
• How does batching improve throughput at the cost of latency? — Amortizes fixed per-batch overhead (network round trip, disk fsync) across multiple items. Each item waits longer (latency increases) but total work per unit time increases.
• What is the USL (Universal Scalability Law) and how does it relate to throughput scaling? — USL extends Amdahl's Law with a retrograde term for coherency overhead. Throughput increases sublinearly, peaks, then decreases as concurrency-induced serialization dominates.
• How would you design a load test to measure both latency and throughput correctly? — Use constant arrival rate (not concurrency-based) to avoid coordinated omission. Measure latency at each percentile under sustained load. Use tools like wrk2 or Gatling that implement open-loop load generation.

Related Questions

• Redis Caching Patterns
• Vertical vs Horizontal Scaling
• API Versioning
• SLOs vs SLAs
• Availability — Five 9s