Is exactly-once delivery possible and how do you achieve it?
Exactly-once delivery means a message is processed by the consumer exactly one time — neither lost nor duplicated. In theory it is achievable; in practice it requires specific infrastructure support and careful application design.
First, clarify the two failure modes. **At-most-once**: the producer sends without waiting for an acknowledgment, or the consumer commits its offset before processing. Messages can be lost but never duplicated. Suitable for metrics or logging. **At-least-once**: the producer retries on timeout and the consumer commits after processing. Messages are never lost but can be processed multiple times on crashes. Most systems default here.
Exactly-once delivery requires solving both sides simultaneously. The producer must not introduce duplicates when it retries, and the consumer must not reprocess a message it already handled.
**Producer side — idempotent producers**: Kafka's idempotent producer assigns each message a sequence number. The broker deduplicates retries within a session using (producer ID, partition, sequence). This eliminates producer-side duplicates but only within a single session — a broker restart resets the producer ID.
**Consumer side — idempotent processing**: The application must make the processing operation idempotent. Common techniques: (1) database upsert keyed on a message ID — inserting the same order twice is a no-op; (2) storing the processed offset alongside the state update in a single atomic transaction; (3) a deduplication table that records seen message IDs with a TTL.
**Kafka transactions** combine both: the producer writes to multiple partitions and commits the consumer offset atomically within a single transaction. Consumers set `isolation.level=read_committed` to skip uncommitted messages. This gives exactly-once semantics within the Kafka ecosystem.
**Distributed transactions (2PC)** can coordinate writes across heterogeneous systems (Kafka + Postgres) but introduce coordinator availability as a bottleneck and are slow. Most teams prefer the outbox pattern instead: write the event to an outbox table in the same DB transaction as the business write, then have a separate process relay it to Kafka. This reduces cross-system coordination to at-least-once + idempotent consumer.
The honest answer: exactly-once is achievable within a single system (Kafka, or a single DB with transactions), but across heterogeneous distributed systems it requires careful design trade-offs, and most production systems settle for at-least-once with idempotent consumers.
Correctly distinguishes at-most-once, at-least-once, and exactly-once; explains idempotent consumers as the practical solution; mentions Kafka transactions or deduplication tables.
Covers producer-side idempotency (Kafka sequence numbers), consumer-side idempotency (atomic offset + state commit), the outbox pattern for cross-system guarantees, and articulates when exactly-once is not worth the complexity.
Reading the answer is step one. Explaining it unprompted — under interview pressure — is what actually matters. Get AI-graded feedback on your answer with follow-up probes on your weak points.
Get Graded — Free Assessment