Kafka — enable.auto.commit Dangers

KafkaSeniorkafka

The Question

What are the dangers of enable.auto.commit in Kafka?

What a Strong Answer Covers

• Both lost + duplicate scenarios
• manual commit fix.

Senior-Level Answer

`enable.auto.commit=true` (the default) instructs the Kafka consumer to automatically commit the current offset to the broker on a fixed interval controlled by `auto.commit.interval.ms` (default 5 seconds). While convenient, this creates two distinct failure modes that are both correctness problems in most production systems.

The first danger is message loss. Auto-commit advances the offset based on a timer, not on whether processing succeeded. If the consumer fetches a batch, the auto-commit fires, and then the consumer crashes before finishing processing, the committed offset has moved forward. On restart, those messages are skipped — silently lost from the consumer's perspective. For any system where missing a message is a bug (financial events, audit logs, state transitions), this is unacceptable.

The second danger is unintended duplicate processing combined with offset amnesia. Suppose processing takes longer than `auto.commit.interval.ms`. The consumer is mid-batch when the commit fires. If the consumer then fails after the commit but before completing downstream writes, it restarts at the committed point — which is past the beginning of the batch but not necessarily past the failed message. Depending on batch boundaries, this can cause inconsistent partial processing.

The safe alternative is manual offset commit: call `commitSync()` or `commitAsync()` only after processing and any downstream writes are complete. `commitSync()` blocks and retries on failure, providing at-least-once semantics with the retry window fully under application control. `commitAsync()` is non-blocking but requires a callback to handle failures — typically used for throughput-sensitive paths where the application can tolerate the last offset being recommitted.

For exactly-once semantics, neither auto nor manual commit alone is sufficient. The pattern is to write both the processed result and the new offset atomically to a transactional datastore — offset commit and business write happen in the same transaction, making them inseparable.

What Separates a 2/3 from a 3/3

Common Mistakes

• Only identifying one failure mode (usually duplicates) and missing the message loss scenario.
• Thinking manual commit eliminates all duplicates — at-least-once is the best you get without transactional writes.
• Not distinguishing commitSync (blocking, retries) from commitAsync (non-blocking, callback-required) trade-offs.
• Believing auto.commit is safe for idempotent consumers — idempotency handles duplicates but not the silent loss scenario.

Follow-Up Questions

• What is the difference between commitSync and commitAsync, and when would you choose each? — commitSync blocks and retries — safe but adds latency. commitAsync is non-blocking — use with a callback; good for high-throughput where occasional missed commits are tolerable.
• How would you implement exactly-once semantics with a Kafka consumer writing to a database? — Store the offset in the same DB transaction as the processed result. On restart, read the last committed offset from the DB and seek the consumer to it.
• If you set auto.commit.interval.ms=100ms, does that make auto-commit safe? — No — the gap still exists between commit and processing completion. You've reduced the window but haven't eliminated the race condition.
• How does Kafka Streams handle offset commits differently from the plain consumer API? — Kafka Streams commits offsets as part of processing checkpoints — tied to state store flushes, giving stronger EOS guarantees without manual commit code.

Related Questions

• Kafka — acks=0/1/all
• Kafka — Why Is It Fast
• Kafka — Consumer Lag
• Kafka — ISR
• Kafka — Consumer Group