Kafka — Rebalance

KafkaSeniorkafka

The Question

What is a Kafka consumer group rebalance?

What a Strong Answer Covers

• At least 3 triggers
• processing stops.

Senior-Level Answer

A consumer group rebalance is the process by which Kafka redistributes partition ownership among the members of a consumer group. It is triggered by membership changes (a consumer joining or leaving, a crash, or a heartbeat timeout), changes in the number of partitions for a subscribed topic, or a subscription change.

During a classic eager rebalance, all consumers revoke their current partition assignments, and then new assignments are computed and distributed. This stop-the-world pause halts consumption for all members — a significant problem for latency-sensitive workloads or large groups. To mitigate this, Kafka introduced cooperative incremental rebalancing (available since 2.4 via `CooperativeStickyAssignor`). In cooperative rebalancing, only the partitions that need to move are revoked and reassigned, allowing other partitions to continue being consumed during the process.

Rebalances are coordinated through the Group Coordinator broker. One consumer in the group is elected as the Group Leader and is responsible for computing the actual partition assignment using the configured `partition.assignment.strategy`. The result is sent back through the coordinator to all members.

From an application standpoint, the critical concern is offset handling during rebalance. A consumer must implement a `ConsumerRebalanceListener` to commit offsets for partitions being revoked (`onPartitionsRevoked`) before they transfer to another consumer. Failure to do this is the most common cause of message reprocessing after a rebalance, because the new owner will start from the last committed offset.

Common causes of excessive rebalancing include: heartbeat timeouts from long processing loops (`max.poll.interval.ms` exceeded), slow consumers removed due to missed polls, and aggressive session timeouts. Tuning `session.timeout.ms`, `heartbeat.interval.ms`, and `max.poll.interval.ms` — alongside switching to the `CooperativeStickyAssignor` — are the primary levers for reducing rebalance frequency and impact.

What Separates a 2/3 from a 3/3

Common Mistakes

• Not knowing that all consumers pause during an eager rebalance — treating it as a background event.
• Skipping offset commits on partition revocation, leading to reprocessing after rebalance.
• Confusing session.timeout.ms (heartbeat-based) with max.poll.interval.ms (poll-based) as rebalance triggers.
• Not knowing that cooperative rebalancing exists or how it differs from eager rebalancing.

Follow-Up Questions

• How does `max.poll.interval.ms` cause rebalances, and how would you fix a consumer hitting it? — Long processing between polls triggers the broker to assume the consumer is dead. Fix by increasing the timeout, reducing batch size, or offloading processing asynchronously.
• What is the difference between `CooperativeStickyAssignor` and `RangeAssignor`? — Cooperative is incremental (only moved partitions revoked); Range is eager with alphabetical-then-round-robin assignment.
• What should a `ConsumerRebalanceListener.onPartitionsRevoked` implementation do? — Commit offsets for the partitions being revoked before they transfer to another consumer.
• How would you reduce rebalance frequency in a high-throughput consumer group? — Increase session.timeout.ms and max.poll.interval.ms, reduce max.poll.records, switch to CooperativeStickyAssignor.

Related Questions

• Kafka — acks=0/1/all
• Kafka — Why Is It Fast
• Kafka — Consumer Lag
• Kafka — ISR
• Kafka — Consumer Group