Why is Kafka fast?
Kafka achieves high throughput—millions of messages per second on commodity hardware—through several design decisions that align with how operating systems and hardware actually work, rather than fighting them.
**Sequential disk I/O**: Kafka writes messages sequentially to an append-only log on disk. Sequential writes are 3-4 orders of magnitude faster than random writes because they avoid seek time on spinning disks and maximise write coalescing on SSDs. Kafka never modifies written data—retention-based deletion removes whole segments, not individual records. This makes Kafka's write path essentially as fast as sequential disk bandwidth allows.
**OS page cache exploitation**: Kafka does not maintain its own in-process message cache. Instead, it relies on the OS page cache—recently written data is kept in RAM automatically by the kernel. Because the producer and consumer typically operate close in time (the "hot path"), consumer reads are often served entirely from the page cache without hitting disk. This is more efficient than a JVM heap cache: the OS page cache survives GC pauses and broker restarts (within the RAM window), and avoids JVM memory management overhead.
**Zero-copy transfer (sendfile)**: When a consumer fetches messages, Kafka uses the `sendfile(2)` system call (or `transferTo` in Java NIO) to transfer data from the file system page cache directly to the network socket, bypassing user space entirely. This eliminates two memory copies and two context switches compared to a read-then-write path. At high throughput, this is a major CPU saving.
**Batching**: Producers batch multiple messages into a single network request, controlled by `linger.ms` and `batch.size`. Brokers write batches as a unit. Consumers fetch batches. Batching amortises network round-trip overhead across many messages, dramatically increasing effective throughput per CPU cycle.
**Compression**: Message batches are compressed (Snappy, LZ4, zstd) at the producer. The compressed batch is stored on the broker and sent to the consumer compressed—the broker never decompresses. This reduces both network and disk I/O, with decompression cost paid only at the consumer.
**Partition parallelism**: Topics are divided into partitions, each an independent ordered log. Producers write to multiple partitions in parallel; consumers in a consumer group each read from a subset of partitions. This scales horizontally—throughput scales with the number of partitions (up to the hardware limit per broker).
Covers sequential I/O, zero-copy sendfile, and batching with accurate technical explanations.
Covers all six mechanisms with accurate low-level detail, explains why page cache is preferred over JVM heap cache, and connects partition parallelism to horizontal scaling.
Reading the answer is step one. Explaining it unprompted — under interview pressure — is what actually matters. Get AI-graded feedback on your answer with follow-up probes on your weak points.
Get Graded — Free Assessment