Kafka — Consumer Group
The Question
How do Kafka consumer groups work?
What a Strong Answer Covers
- one partition → one consumer per group
- "idle if more consumers than partitions
Senior-Level Answer
A Kafka consumer group is a set of consumers that collaboratively consume messages from one or more topics. The core rule is that each partition in a topic is assigned to exactly one consumer within a group at any given time. This means a group achieves parallelism up to the number of partitions — adding more consumers than partitions leaves some consumers idle.
Group membership and partition assignment are coordinated by the Group Coordinator, a broker elected for that group. When a consumer joins or leaves, the coordinator triggers a rebalance to redistribute partitions. Each group independently tracks its own offsets, stored in the internal `__consumer_offsets` topic. This allows multiple groups to consume the same topic at different rates without interference — a common pattern for fan-out (e.g., one group for real-time processing, another for audit logging).
Offset commits are the mechanism by which a consumer records progress. With `enable.auto.commit=true`, offsets are committed on a timer, which creates at-least-once semantics with a window of potential reprocessing on crash. For stronger guarantees, manual commit (`commitSync` or `commitAsync`) lets consumers commit only after processing is confirmed — or after writing to a downstream system transactionally.
Consumer group state is visible via `kafka-consumer-groups.sh --describe`, which reports partition assignments, current offsets, log-end offsets, and lag per partition. Monitoring lag is the primary operational concern — sustained growing lag signals that consumer throughput is insufficient and scaling or optimization is needed.
A single consumer with `group.id` set to a unique value effectively acts as a broadcast consumer, receiving all partitions for itself — the standard pattern when you need every message on every instance (e.g., cache invalidation).
What Separates a 2/3 from a 3/3
2/3 — Passing but Incomplete
Explains one-partition-to-one-consumer assignment, mentions that offset tracking is per group, and understands that multiple groups can consume independently.
3/3 — Strong Answer
Covers partition assignment limits (consumers > partitions = idle consumers), offset commit semantics (auto vs. manual), and operational monitoring via consumer lag. Mentions the group coordinator's role in rebalancing.
Common Mistakes
- Claiming messages are load-balanced across consumers at the message level — parallelism is at the partition level.
- Not knowing that adding consumers beyond the partition count has no throughput benefit.
- Confusing group offsets with partition offsets — each group has its own committed offset per partition.
- Forgetting that two different consumer groups each receive all messages independently.
Follow-Up Questions
- A topic has 6 partitions and your consumer group has 8 consumers. What happens to the extra 2 consumers? — They sit idle — no partition to assign. Understand the ceiling on parallelism.
- How would you implement exactly-once processing with a Kafka consumer group? — Transactional producers + idempotent consumers, or Kafka Streams with EOS, or manual offset commit inside a DB transaction.
- What is consumer lag and how do you act on it? — Lag = log end offset minus committed offset. Sustained growth means scaling consumers, optimizing processing, or increasing partitions.
- How does a new consumer joining a group get its partition assignment? — Triggers rebalance — group coordinator + elected leader consumer run the assignor strategy (range, round-robin, sticky).
Related Questions
Can You Explain This Cold?
Reading the answer is step one. Explaining it unprompted — under interview pressure — is what actually matters. Get AI-graded feedback on your answer with follow-up probes on your weak points.
Get Graded — Free Assessment