Kafka — Topic, Partition, Offset

KafkaEntrydbkafka

The Question

Explain Kafka topics, partitions, and offsets.

What a Strong Answer Covers

• partition = parallelism
• "offset = sequential per partition
• "ordering within partition only

Senior-Level Answer

A Kafka topic is a named, durable, append-only log that producers write to and consumers read from. Topics are the primary organizational unit — you create a topic for each logical stream of data (e.g., `user-events`, `order-updates`).

A topic is divided into one or more partitions. Each partition is an independent, ordered, immutable log stored on a broker's disk. Partitions are the unit of parallelism in Kafka: multiple consumers in a consumer group can read different partitions concurrently, and multiple brokers can host different partitions of the same topic, enabling horizontal scaling. The partition count also sets the maximum parallelism ceiling for any consumer group reading that topic.

Messages within a partition are assigned a monotonically increasing integer called an offset. Offsets are per-partition and always increase — Kafka never reuses or modifies them. The offset is the consumer's cursor into a partition: by committing an offset, a consumer records that it has processed all messages up to that point. On restart, it resumes from the committed offset.

Ordering in Kafka is guaranteed within a partition, not across partitions. If you need all events for a specific entity (e.g., all orders for customer_id=42) to be processed in order, you must route them to the same partition by using a consistent partition key. By default, the producer hashes the message key to select a partition — messages with the same key always go to the same partition.

Messages are retained in Kafka for a configurable period (or until a size limit is hit) regardless of whether they've been consumed. This means consumers can re-read historical data by seeking to an earlier offset, and multiple consumer groups can independently consume the same topic at different paces without affecting each other.

What Separates a 2/3 from a 3/3

Common Mistakes

• Claiming Kafka guarantees ordering across partitions — ordering is strictly per-partition only.
• Confusing offset with a message ID — offset is a positional cursor within a partition, not a globally unique identifier.
• Not knowing that partition count is the parallelism ceiling for consumer groups.
• Assuming a message is deleted after consumption — Kafka retains messages based on time/size policy, not consumption status.

Follow-Up Questions

• Why does partition key selection matter for ordering guarantees? — Messages with the same key route to the same partition — only then can you guarantee ordered delivery for that key's events.
• Your topic has 3 partitions and your consumer group has 5 consumers. What happens? — Two consumers sit idle — Kafka only assigns one consumer per partition per group. Parallelism is capped at partition count.
• Can you reduce the number of partitions in an existing Kafka topic? — No — Kafka does not support partition reduction. You can increase partitions, but existing messages keep their old partition assignments, which can break key-based ordering for existing data.
• How would you re-process all events for a topic from the beginning? — Create a new consumer group (which starts with no committed offsets) and set auto.offset.reset=earliest, or use kafka-consumer-groups.sh to reset offsets to the beginning.

Related Questions

• Kafka — acks=0/1/all
• Kafka — Why Is It Fast
• Kafka — Consumer Lag
• Kafka — ISR
• Kafka — Consumer Group