Kafka — Consumer Lag

KafkaSeniorkafka

The Question

What is Kafka consumer lag and how do you monitor it?

What a Strong Answer Covers

• latest offset minus consumer offset
• monitoring tool.

Senior-Level Answer

Kafka consumer lag is the difference between the latest offset produced to a partition (log-end offset, LEO) and the last offset committed by the consumer group for that partition. It measures how far behind the consumer is from the producer—how many unconsumed messages are waiting.

**How offsets work**: Each partition in Kafka has a monotonically increasing offset. The producer advances the log-end offset with every written message. The consumer reads messages and periodically commits its current offset to the `__consumer_offsets` internal topic. Lag = LEO - committed offset. If the consumer stops or processes messages slowly, lag grows.

**Why lag matters**: A growing lag means the consumer cannot keep up with the producer. In real-time pipelines this means increasing processing delay—messages are being processed later than they were produced. A lag that grows without bound eventually causes backpressure issues, OOM if messages are buffered in memory, or SLA violations. A lag spike after a deployment or scaling event is expected; lag that grows monotonically is a reliability signal.

**How to measure it**: - **kafka-consumer-groups.sh --describe**: CLI tool that outputs group ID, topic, partition, current offset, log-end offset, and lag per partition. Direct and authoritative. - **Kafka JMX metrics**: Brokers expose `records-lag-max` and `records-lag` per consumer group and partition via JMX. Scraped by Prometheus via the JMX Exporter. - **Burrow (LinkedIn)**: Purpose-built lag monitor that analyses lag trends over a sliding window rather than point-in-time values—distinguishes between a consumer that is slow but catching up vs. one that is falling further behind. This is a more actionable signal than raw lag number. - **Confluent Control Center / AWS MSK Monitoring / Datadog Kafka integration**: Managed monitoring solutions for cloud-hosted Kafka.

**Alerting on lag**: Alert on sustained lag growth rate, not absolute lag. A large but stable lag may be fine (batch processor); a small but growing lag is an early warning. Set alert thresholds based on the acceptable processing delay for the use case.

**Remediation**: Increase consumer parallelism by adding consumer instances (up to partition count). Optimize consumer processing (reduce per-message cost). Increase `fetch.min.bytes` / `max.poll.records` for throughput-oriented consumers. If the consumer is persistently unable to catch up, consider adding partitions and rebalancing (requires careful ordering considerations).

What Separates a 2/3 from a 3/3

Common Mistakes

• Defining lag as 'number of unread messages' without specifying it's per partition and per consumer group.
• Alerting on absolute lag value without considering whether it's growing or stable.
• Not knowing that lag can only be reduced by adding consumers up to the partition count—beyond that, adding consumers has no effect.
• Confusing the committed offset with the current fetch offset—lag is measured against committed (durable) offsets.

Follow-Up Questions

• If you add more consumer instances than partitions, will lag decrease faster? — No. Each partition is consumed by exactly one consumer in a group. Extra consumers beyond partition count are idle. To scale further, increase partition count first.
• How does Burrow's window-based lag analysis improve on simple lag metrics? — A point-in-time lag of 10k could be fine (stable) or critical (growing). Burrow evaluates the slope over a window—it distinguishes 'catching up', 'stable', and 'falling behind' states.
• What is the difference between the current offset and the committed offset in Kafka? — Current offset: where the consumer is actively reading (in-flight). Committed offset: last durably checkpointed position. Lag is measured against committed, not current—uncommitted progress is invisible to lag metrics.
• How would you investigate a sudden consumer lag spike after a deployment? — Check consumer group status for rebalancing (consumers joining/leaving cause pause). Check processing time per message for regressions. Verify new code isn't blocking the poll loop.

Related Questions

• Kafka — acks=0/1/all
• Kafka — Why Is It Fast
• Kafka — ISR
• Kafka — Consumer Group
• Kafka — Rebalance