← Back to Distributed Systems

Exactly-once Delivery

Distributed SystemsSenior

The Question

Is exactly-once delivery possible and how do you achieve it?

What a Strong Answer Covers

  • at-least-once + idempotency key
  • Kafka mechanism.

Senior-Level Answer

Exactly-once delivery means a message is processed by the consumer exactly one time — neither lost nor duplicated. In theory it is achievable; in practice it requires specific infrastructure support and careful application design.

First, clarify the two failure modes. **At-most-once**: the producer sends without waiting for an acknowledgment, or the consumer commits its offset before processing. Messages can be lost but never duplicated. Suitable for metrics or logging. **At-least-once**: the producer retries on timeout and the consumer commits after processing. Messages are never lost but can be processed multiple times on crashes. Most systems default here.

Exactly-once delivery requires solving both sides simultaneously. The producer must not introduce duplicates when it retries, and the consumer must not reprocess a message it already handled.

**Producer side — idempotent producers**: Kafka's idempotent producer assigns each message a sequence number. The broker deduplicates retries within a session using (producer ID, partition, sequence). This eliminates producer-side duplicates but only within a single session — a broker restart resets the producer ID.

**Consumer side — idempotent processing**: The application must make the processing operation idempotent. Common techniques: (1) database upsert keyed on a message ID — inserting the same order twice is a no-op; (2) storing the processed offset alongside the state update in a single atomic transaction; (3) a deduplication table that records seen message IDs with a TTL.

**Kafka transactions** combine both: the producer writes to multiple partitions and commits the consumer offset atomically within a single transaction. Consumers set `isolation.level=read_committed` to skip uncommitted messages. This gives exactly-once semantics within the Kafka ecosystem.

**Distributed transactions (2PC)** can coordinate writes across heterogeneous systems (Kafka + Postgres) but introduce coordinator availability as a bottleneck and are slow. Most teams prefer the outbox pattern instead: write the event to an outbox table in the same DB transaction as the business write, then have a separate process relay it to Kafka. This reduces cross-system coordination to at-least-once + idempotent consumer.

The honest answer: exactly-once is achievable within a single system (Kafka, or a single DB with transactions), but across heterogeneous distributed systems it requires careful design trade-offs, and most production systems settle for at-least-once with idempotent consumers.

What Separates a 2/3 from a 3/3

2/3 — Passing but Incomplete

Correctly distinguishes at-most-once, at-least-once, and exactly-once; explains idempotent consumers as the practical solution; mentions Kafka transactions or deduplication tables.

3/3 — Strong Answer

Covers producer-side idempotency (Kafka sequence numbers), consumer-side idempotency (atomic offset + state commit), the outbox pattern for cross-system guarantees, and articulates when exactly-once is not worth the complexity.

Common Mistakes

  • Claiming exactly-once is simply impossible — it is possible within constrained systems.
  • Conflating idempotency with exactly-once — idempotency makes at-least-once safe but does not prevent reprocessing.
  • Not explaining what happens to the offset commit when the consumer crashes between processing and commit.
  • Ignoring the outbox pattern when asked about cross-database and message-broker scenarios.

Follow-Up Questions

  • How does the outbox pattern work and what problem does it solve? — A DB transaction writes the event to an outbox table atomically with the business change; a poller or CDC reads the outbox and publishes to the broker.
  • What is the difference between idempotent producers and Kafka transactions? — Idempotent producers deduplicate within a partition; transactions extend this to atomic multi-partition writes and offset commits.
  • How would you build exactly-once semantics for a payment processing service without Kafka transactions? — Deduplication table keyed on payment_id with a unique constraint; upsert instead of insert; return the existing result on duplicate.
  • What is isolation.level=read_committed in Kafka and why does it matter for exactly-once consumers? — Without it, consumers read messages from aborted transactions, breaking the guarantee.

Related Questions

  • Consistent Hashing
  • Rate Limiting
  • CDN
  • Load Balancer — L4 vs L7
  • Distributed Transactions — 2PC, Saga

Can You Explain This Cold?

Reading the answer is step one. Explaining it unprompted — under interview pressure — is what actually matters. Get AI-graded feedback on your answer with follow-up probes on your weak points.

Get Graded — Free Assessment
GrindQuestionsAITechnical interview assessment
TermsPrivacyAbout