Question 1

What do acks=0, acks=1, and acks=all mean in Kafka?

Accepted Answer

Kafka's `acks` producer configuration controls how many broker acknowledgements the producer requires before considering a write successful. It is the primary knob for trading off durability against throughput and latency.

**acks=0 (fire-and-forget)**: The producer sends the message and does not wait for any acknowledgement from the broker. The message is written to the network buffer and the producer immediately continues. This provides the absolute lowest latency and highest throughput, but offers zero durability guarantee—if the broker crashes before writing to disk, the message is silently lost. Appropriate for high-volume metrics or telemetry where occasional data loss is acceptable.

**acks=1 (leader ack)**: The producer waits for acknowledgement from the partition leader only. The leader writes the message to its local log and responds. This is the default in many Kafka client libraries. It provides a reasonable durability guarantee under normal conditions, but there is a data loss window: if the leader crashes after acknowledging but before the message is replicated to followers, a newly elected follower will not have the message and it will be lost. Latency is one round trip to the leader. Suitable for use cases where occasional loss on leader failure is tolerable.

**acks=all (acks=-1)**: The producer waits for the partition leader to receive acknowledgements from all in-sync replicas (ISR) before responding. The ISR is the set of replicas that are fully caught up with the leader within `replica.lag.time.max.ms`. A message acknowledged with acks=all is durable against any single broker failure (and against multiple failures if the ISR has sufficient members). This configuration must be paired with `min.insync.replicas` (broker/topic config) to prevent the ISR from shrinking to just the leader—setting `min.insync.replicas=2` with a replication factor of 3 ensures at least two brokers must acknowledge. Latency is one round trip plus the replication lag to the slowest ISR member. Required for financial transactions, order processing, and any use case where message loss is unacceptable.

**Interaction with retries and idempotency**: With acks=1 or acks=all, network timeouts can cause duplicate messages if the producer retries a message the broker already committed. Setting `enable.idempotence=true` (Kafka 0.11+) with `acks=all` enables exactly-once semantics at the producer level—the broker deduplicates using a sequence number per producer session.

Question 2

Why is Kafka fast?

Accepted Answer

Kafka achieves high throughput—millions of messages per second on commodity hardware—through several design decisions that align with how operating systems and hardware actually work, rather than fighting them.

**Sequential disk I/O**: Kafka writes messages sequentially to an append-only log on disk. Sequential writes are 3-4 orders of magnitude faster than random writes because they avoid seek time on spinning disks and maximise write coalescing on SSDs. Kafka never modifies written data—retention-based deletion removes whole segments, not individual records. This makes Kafka's write path essentially as fast as sequential disk bandwidth allows.

**OS page cache exploitation**: Kafka does not maintain its own in-process message cache. Instead, it relies on the OS page cache—recently written data is kept in RAM automatically by the kernel. Because the producer and consumer typically operate close in time (the "hot path"), consumer reads are often served entirely from the page cache without hitting disk. This is more efficient than a JVM heap cache: the OS page cache survives GC pauses and broker restarts (within the RAM window), and avoids JVM memory management overhead.

**Zero-copy transfer (sendfile)**: When a consumer fetches messages, Kafka uses the `sendfile(2)` system call (or `transferTo` in Java NIO) to transfer data from the file system page cache directly to the network socket, bypassing user space entirely. This eliminates two memory copies and two context switches compared to a read-then-write path. At high throughput, this is a major CPU saving.

**Batching**: Producers batch multiple messages into a single network request, controlled by `linger.ms` and `batch.size`. Brokers write batches as a unit. Consumers fetch batches. Batching amortises network round-trip overhead across many messages, dramatically increasing effective throughput per CPU cycle.

**Compression**: Message batches are compressed (Snappy, LZ4, zstd) at the producer. The compressed batch is stored on the broker and sent to the consumer compressed—the broker never decompresses. This reduces both network and disk I/O, with decompression cost paid only at the consumer.

**Partition parallelism**: Topics are divided into partitions, each an independent ordered log. Producers write to multiple partitions in parallel; consumers in a consumer group each read from a subset of partitions. This scales horizontally—throughput scales with the number of partitions (up to the hardware limit per broker).

Question 3

What is Kafka consumer lag and how do you monitor it?

Accepted Answer

Kafka consumer lag is the difference between the latest offset produced to a partition (log-end offset, LEO) and the last offset committed by the consumer group for that partition. It measures how far behind the consumer is from the producer—how many unconsumed messages are waiting.

**How offsets work**: Each partition in Kafka has a monotonically increasing offset. The producer advances the log-end offset with every written message. The consumer reads messages and periodically commits its current offset to the `__consumer_offsets` internal topic. Lag = LEO - committed offset. If the consumer stops or processes messages slowly, lag grows.

**Why lag matters**: A growing lag means the consumer cannot keep up with the producer. In real-time pipelines this means increasing processing delay—messages are being processed later than they were produced. A lag that grows without bound eventually causes backpressure issues, OOM if messages are buffered in memory, or SLA violations. A lag spike after a deployment or scaling event is expected; lag that grows monotonically is a reliability signal.

**How to measure it**:
- **kafka-consumer-groups.sh --describe**: CLI tool that outputs group ID, topic, partition, current offset, log-end offset, and lag per partition. Direct and authoritative.
- **Kafka JMX metrics**: Brokers expose `records-lag-max` and `records-lag` per consumer group and partition via JMX. Scraped by Prometheus via the JMX Exporter.
- **Burrow (LinkedIn)**: Purpose-built lag monitor that analyses lag trends over a sliding window rather than point-in-time values—distinguishes between a consumer that is slow but catching up vs. one that is falling further behind. This is a more actionable signal than raw lag number.
- **Confluent Control Center / AWS MSK Monitoring / Datadog Kafka integration**: Managed monitoring solutions for cloud-hosted Kafka.

**Alerting on lag**: Alert on sustained lag growth rate, not absolute lag. A large but stable lag may be fine (batch processor); a small but growing lag is an early warning. Set alert thresholds based on the acceptable processing delay for the use case.

**Remediation**: Increase consumer parallelism by adding consumer instances (up to partition count). Optimize consumer processing (reduce per-message cost). Increase `fetch.min.bytes` / `max.poll.records` for throughput-oriented consumers. If the consumer is persistently unable to catch up, consider adding partitions and rebalancing (requires careful ordering considerations).

Question 4

What is the ISR (In-Sync Replicas) in Kafka?

Accepted Answer

The In-Sync Replicas (ISR) list is a dynamic set maintained per partition by the Kafka broker acting as partition leader. A replica is considered "in-sync" if it has fetched all messages from the leader within the window defined by `replica.lag.time.max.ms` (default 30s). If a follower falls behind — due to network issues, GC pauses, or slow disk — it is removed from the ISR.

The ISR is critical for durability guarantees. When a producer sends with `acks=all` (or `acks=-1`), the leader waits until every replica in the current ISR has acknowledged the message before returning success to the producer. This means ISR shrinkage directly impacts write latency — a slow replica that stays in ISR causes the producer to wait for it.

During leader election, Kafka by default only allows replicas in the ISR to be elected as the new leader. This prevents data loss: a non-ISR replica may be missing messages the previous leader had already acknowledged. The `unclean.leader.election.enable` setting (default false) controls whether out-of-ISR replicas can be elected — enabling it trades consistency for availability.

Operationally, you can inspect ISR status via `kafka-topics.sh --describe` or through broker metrics (`UnderReplicatedPartitions`, `IsrShrinksPerSec`). A consistently shrinking ISR is a leading indicator of broker health issues and should trigger investigation before it affects availability.

The interplay between ISR, `min.insync.replicas`, and `acks` defines your actual durability contract. ISR is the mechanism; the other two settings are the policy knobs that tell Kafka how to enforce that contract under degraded conditions.

Question 5

How do Kafka consumer groups work?

Accepted Answer

A Kafka consumer group is a set of consumers that collaboratively consume messages from one or more topics. The core rule is that each partition in a topic is assigned to exactly one consumer within a group at any given time. This means a group achieves parallelism up to the number of partitions — adding more consumers than partitions leaves some consumers idle.

Group membership and partition assignment are coordinated by the Group Coordinator, a broker elected for that group. When a consumer joins or leaves, the coordinator triggers a rebalance to redistribute partitions. Each group independently tracks its own offsets, stored in the internal `__consumer_offsets` topic. This allows multiple groups to consume the same topic at different rates without interference — a common pattern for fan-out (e.g., one group for real-time processing, another for audit logging).

Offset commits are the mechanism by which a consumer records progress. With `enable.auto.commit=true`, offsets are committed on a timer, which creates at-least-once semantics with a window of potential reprocessing on crash. For stronger guarantees, manual commit (`commitSync` or `commitAsync`) lets consumers commit only after processing is confirmed — or after writing to a downstream system transactionally.

Consumer group state is visible via `kafka-consumer-groups.sh --describe`, which reports partition assignments, current offsets, log-end offsets, and lag per partition. Monitoring lag is the primary operational concern — sustained growing lag signals that consumer throughput is insufficient and scaling or optimization is needed.

A single consumer with `group.id` set to a unique value effectively acts as a broadcast consumer, receiving all partitions for itself — the standard pattern when you need every message on every instance (e.g., cache invalidation).

Question 6

What is a Kafka consumer group rebalance?

Accepted Answer

A consumer group rebalance is the process by which Kafka redistributes partition ownership among the members of a consumer group. It is triggered by membership changes (a consumer joining or leaving, a crash, or a heartbeat timeout), changes in the number of partitions for a subscribed topic, or a subscription change.

During a classic eager rebalance, all consumers revoke their current partition assignments, and then new assignments are computed and distributed. This stop-the-world pause halts consumption for all members — a significant problem for latency-sensitive workloads or large groups. To mitigate this, Kafka introduced cooperative incremental rebalancing (available since 2.4 via `CooperativeStickyAssignor`). In cooperative rebalancing, only the partitions that need to move are revoked and reassigned, allowing other partitions to continue being consumed during the process.

Rebalances are coordinated through the Group Coordinator broker. One consumer in the group is elected as the Group Leader and is responsible for computing the actual partition assignment using the configured `partition.assignment.strategy`. The result is sent back through the coordinator to all members.

From an application standpoint, the critical concern is offset handling during rebalance. A consumer must implement a `ConsumerRebalanceListener` to commit offsets for partitions being revoked (`onPartitionsRevoked`) before they transfer to another consumer. Failure to do this is the most common cause of message reprocessing after a rebalance, because the new owner will start from the last committed offset.

Common causes of excessive rebalancing include: heartbeat timeouts from long processing loops (`max.poll.interval.ms` exceeded), slow consumers removed due to missed polls, and aggressive session timeouts. Tuning `session.timeout.ms`, `heartbeat.interval.ms`, and `max.poll.interval.ms` — alongside switching to the `CooperativeStickyAssignor` — are the primary levers for reducing rebalance frequency and impact.

Question 7

What does min.insync.replicas control in Kafka?

Accepted Answer

`min.insync.replicas` (commonly abbreviated `min.isr`) is a broker/topic-level configuration that specifies the minimum number of replicas that must be in the ISR for a produce request with `acks=all` to be accepted. If the current ISR falls below this threshold, the broker rejects produce requests with a `NotEnoughReplicasException`, effectively halting writes rather than accepting messages that could be lost.

The canonical safe configuration for a 3-broker cluster is `replication.factor=3` with `min.insync.replicas=2`. This means you can tolerate one broker failure and still accept writes — the remaining 2 replicas satisfy the minimum. Setting `min.insync.replicas=3` requires all three replicas to be in sync, meaning any single broker failure stops writes entirely. Setting it to 1 provides no durability benefit over `acks=1`.

This setting only takes effect when the producer uses `acks=all`. With `acks=1` or `acks=0`, the broker ignores `min.insync.replicas` entirely — a common source of confusion. The setting is purely a server-side guard for the strongest producer acknowledgment mode.

The trade-off is availability versus durability: a lower `min.insync.replicas` keeps writes flowing under degraded conditions but risks data loss if the leader fails before lagging replicas catch up. A higher value maximizes durability but reduces the failure tolerance window. In practice, `replication.factor=3, min.insync.replicas=2, acks=all` is the industry-standard baseline for durable Kafka deployments, giving you a one-failure tolerance with no data loss.

The setting can be configured at the broker level (`min.insync.replicas` in `server.properties`) as a default, and overridden per-topic via `kafka-configs.sh`, allowing you to apply stricter durability to critical topics without impacting higher-throughput, lower-durability topics.

Question 8

What does auto.offset.reset do in Kafka?

Accepted Answer

`auto.offset.reset` is a consumer configuration that determines where Kafka begins reading messages when a consumer group has no committed offset for a partition — or when the committed offset is out of range (e.g., it was deleted due to log retention). The two primary values are `earliest` and `latest`.

With `earliest`, the consumer starts from the oldest available message on the partition — the beginning of the retained log. This is appropriate for new consumers that need to process all historical data, such as a freshly deployed service reading an event log to rebuild state. With `latest`, the consumer starts from the current end of the log, receiving only messages produced after the consumer first started. This is typically used for real-time feeds where historical data is irrelevant.

The third option, `none`, causes the consumer to throw an exception if no committed offset is found, forcing you to handle the bootstrap case explicitly in code. It is rarely used in production but can be valuable in strict operational environments where starting from an unexpected offset would be a bug.

Critically, `auto.offset.reset` only applies when there is no valid committed offset for the group. If a committed offset exists — even a stale one — Kafka uses it and ignores this setting entirely. This trips up engineers who reset a consumer group or deploy a new group expecting a specific behavior but forget that existing committed offsets take precedence.

A common production scenario where this matters: your retention policy deletes log segments, and a lagging consumer's last committed offset no longer exists in the log. The consumer's offset is now out of range. With `earliest`, it resets to the beginning of available data (potential reprocessing of recent messages); with `latest`, it skips ahead to current (potential data loss). Knowing which is appropriate for your use case is an architectural decision, not just a configuration detail.

Question 9

What are the dangers of enable.auto.commit in Kafka?

Accepted Answer

`enable.auto.commit=true` (the default) instructs the Kafka consumer to automatically commit the current offset to the broker on a fixed interval controlled by `auto.commit.interval.ms` (default 5 seconds). While convenient, this creates two distinct failure modes that are both correctness problems in most production systems.

The first danger is message loss. Auto-commit advances the offset based on a timer, not on whether processing succeeded. If the consumer fetches a batch, the auto-commit fires, and then the consumer crashes before finishing processing, the committed offset has moved forward. On restart, those messages are skipped — silently lost from the consumer's perspective. For any system where missing a message is a bug (financial events, audit logs, state transitions), this is unacceptable.

The second danger is unintended duplicate processing combined with offset amnesia. Suppose processing takes longer than `auto.commit.interval.ms`. The consumer is mid-batch when the commit fires. If the consumer then fails after the commit but before completing downstream writes, it restarts at the committed point — which is past the beginning of the batch but not necessarily past the failed message. Depending on batch boundaries, this can cause inconsistent partial processing.

The safe alternative is manual offset commit: call `commitSync()` or `commitAsync()` only after processing and any downstream writes are complete. `commitSync()` blocks and retries on failure, providing at-least-once semantics with the retry window fully under application control. `commitAsync()` is non-blocking but requires a callback to handle failures — typically used for throughput-sensitive paths where the application can tolerate the last offset being recommitted.

For exactly-once semantics, neither auto nor manual commit alone is sufficient. The pattern is to write both the processed result and the new offset atomically to a transactional datastore — offset commit and business write happen in the same transaction, making them inseparable.

Question 10

What is a Kafka idempotent producer and why does it matter?

Accepted Answer

A Kafka idempotent producer is a producer configured with `enable.idempotence=true` that guarantees exactly-once delivery to a single partition within a single producer session. Without idempotence, when a producer retries a failed send (due to a timeout or transient network error), the broker may have already written the message but failed to acknowledge it — resulting in a duplicate.

The mechanism works through two identifiers. The broker assigns each producer a unique Producer ID (PID) on initialization. Each message batch sent by that producer carries a monotonically increasing sequence number per partition. When the broker receives a batch, it checks the sequence number: if the batch was already written (sequence number already seen), the broker discards it and returns success without writing again. If the sequence number is out of order (gap detected), the broker returns an `OutOfOrderSequenceException`.

Enabling idempotence automatically forces `acks=all`, `retries=Integer.MAX_VALUE`, and `max.in.flight.requests.per.connection` to at most 5. These constraints are required to preserve ordering: with `acks=all`, the broker only acknowledges after durably writing; with bounded in-flight requests and sequence numbers, the broker can detect and reject out-of-order duplicates from retries.

Idempotence has important scope limits. It deduplicates within a single producer session — if the producer process restarts, it gets a new PID, and the broker has no way to correlate the new producer with the old one. It also only covers a single partition. For multi-partition atomic writes (e.g., writing to multiple topics transactionally), you need Kafka transactions (`transactional.id` + `initTransactions()` + `commitTransaction()`), which build on top of idempotent producers.

In practice, enabling idempotence is a low-cost correctness improvement with no application-level code changes required — it should be the default for any producer where duplicate messages would cause problems.

Question 11

Explain Kafka topics, partitions, and offsets.

Accepted Answer

A Kafka topic is a named, durable, append-only log that producers write to and consumers read from. Topics are the primary organizational unit — you create a topic for each logical stream of data (e.g., `user-events`, `order-updates`).

A topic is divided into one or more partitions. Each partition is an independent, ordered, immutable log stored on a broker's disk. Partitions are the unit of parallelism in Kafka: multiple consumers in a consumer group can read different partitions concurrently, and multiple brokers can host different partitions of the same topic, enabling horizontal scaling. The partition count also sets the maximum parallelism ceiling for any consumer group reading that topic.

Messages within a partition are assigned a monotonically increasing integer called an offset. Offsets are per-partition and always increase — Kafka never reuses or modifies them. The offset is the consumer's cursor into a partition: by committing an offset, a consumer records that it has processed all messages up to that point. On restart, it resumes from the committed offset.

Ordering in Kafka is guaranteed within a partition, not across partitions. If you need all events for a specific entity (e.g., all orders for customer_id=42) to be processed in order, you must route them to the same partition by using a consistent partition key. By default, the producer hashes the message key to select a partition — messages with the same key always go to the same partition.

Messages are retained in Kafka for a configurable period (or until a size limit is hit) regardless of whether they've been consumed. This means consumers can re-read historical data by seeking to an earlier offset, and multiple consumer groups can independently consume the same topic at different paces without affecting each other.

Kafka Interview Questions — What Senior Engineers Need to Know

All 11 Questions

How to Prepare

Related Topics

Test Your Knowledge