Kafka — Idempotent Producer

KafkaSeniorkafka

The Question

What is a Kafka idempotent producer and why does it matter?

What a Strong Answer Covers

• Sequence number
• broker dedup
• retries as the problem.

Senior-Level Answer

A Kafka idempotent producer is a producer configured with `enable.idempotence=true` that guarantees exactly-once delivery to a single partition within a single producer session. Without idempotence, when a producer retries a failed send (due to a timeout or transient network error), the broker may have already written the message but failed to acknowledge it — resulting in a duplicate.

The mechanism works through two identifiers. The broker assigns each producer a unique Producer ID (PID) on initialization. Each message batch sent by that producer carries a monotonically increasing sequence number per partition. When the broker receives a batch, it checks the sequence number: if the batch was already written (sequence number already seen), the broker discards it and returns success without writing again. If the sequence number is out of order (gap detected), the broker returns an `OutOfOrderSequenceException`.

Enabling idempotence automatically forces `acks=all`, `retries=Integer.MAX_VALUE`, and `max.in.flight.requests.per.connection` to at most 5. These constraints are required to preserve ordering: with `acks=all`, the broker only acknowledges after durably writing; with bounded in-flight requests and sequence numbers, the broker can detect and reject out-of-order duplicates from retries.

Idempotence has important scope limits. It deduplicates within a single producer session — if the producer process restarts, it gets a new PID, and the broker has no way to correlate the new producer with the old one. It also only covers a single partition. For multi-partition atomic writes (e.g., writing to multiple topics transactionally), you need Kafka transactions (`transactional.id` + `initTransactions()` + `commitTransaction()`), which build on top of idempotent producers.

In practice, enabling idempotence is a low-cost correctness improvement with no application-level code changes required — it should be the default for any producer where duplicate messages would cause problems.

What Separates a 2/3 from a 3/3

Common Mistakes

• Claiming idempotent producers provide exactly-once across restarts — the PID resets on producer restart, so cross-session deduplication is not guaranteed.
• Confusing idempotent producer with Kafka transactions — idempotence is per-partition; transactions are for multi-partition atomicity.
• Not knowing that enabling idempotence automatically changes acks, retries, and max.in.flight settings.
• Thinking idempotence eliminates the need for consumer-side deduplication — it only covers the producer-to-broker path.

Follow-Up Questions

• An idempotent producer restarts due to a crash. Does it still prevent duplicates? — No — it gets a new PID on restart. Any unacknowledged messages from the previous session may be re-sent and treated as new by the broker.
• When would you use Kafka transactions instead of just an idempotent producer? — When you need atomic writes across multiple partitions or topics — e.g., consume-transform-produce pipelines where partial writes would be incorrect.
• Why is max.in.flight.requests.per.connection capped at 5 when idempotence is enabled? — With more in-flight batches, the broker cannot guarantee that sequence number gaps from reordered retries are actual duplicates vs. real ordering violations.
• What exception does a broker return if it receives a batch with an unexpected sequence number? — OutOfOrderSequenceException — indicates a sequencing gap that the broker cannot safely deduplicate.

Related Questions

• Kafka — acks=0/1/all
• Kafka — Why Is It Fast
• Kafka — Consumer Lag
• Kafka — ISR
• Kafka — Consumer Group