Kafka exactly-once semantics: idempotent producer and transactions

A Kafka Streams job enriches order events and writes results to an output topic. During a broker rolling restart, the job retries a batch already written, producing 3,000 duplicate output records downstream. The fix was one config line: enable.idempotence=true. The feature existed since 2017. Nobody had turned it on.

Layer 1 — Idempotent producer (KIP-98, Kafka 0.11.0)

The idempotent producer eliminates duplicates caused by producer retries to the broker.

On startup, the producer is assigned a unique producer-ID by the broker. Every message sent to a partition carries a monotonically increasing sequence number scoped to that producer-ID and partition. The broker tracks the last-seen sequence number per (producer-ID, partition) pair.

When a producer retries — because the network dropped the ack — the broker sees the same (producer-ID, sequence) again. It recognises the duplicate, silently acks it, and discards the write. Out-of-order sequences (possible if a retry arrives late) are rejected.

• Enabled by: enable.idempotence=true
• Overhead: ~3% throughput
• Available since: Kafka 0.11.0 (mid-2017)

This solves Leg 1 duplicates only. It does not help if the consumer crashes after processing.

Layer 2 — Transactional producer (KIP-98)

Transactions extend exactly-once to the full read-process-write pipeline within Kafka.

A transactional producer wraps a batch of writes across multiple partitions plus a consumer-offset commit into one atomic unit managed by a transaction coordinator broker. Either all writes and the offset commit become visible to downstream consumers, or none do.

Typical Kafka Streams pattern:

beginTransaction()
  consume from input-partition P0 at offset 42
  produce to output-partition P1
  sendOffsetsToTransaction(group-id, {P0: offset 43})
commitTransaction()

If the job crashes mid-flight, the transaction aborts on restart. The input offset is not advanced. The partial output is rolled back (consumers with isolation.level=read_committed skip aborted records). The job reprocesses from offset 42 — idempotent producer deduplicates the retry.

• Enabled by: transactional.id=my-app-v1
• Consumer must set: isolation.level=read_committed
• Overhead: ~20–30% throughput (two-phase commit between coordinator and partition leaders)

What Kafka transactions do NOT cover

Kafka transactions are atomic within Kafka. The moment you write to Postgres or call Stripe, you leave the transaction boundary. If the Kafka offset commits but the Postgres write fails (or vice versa), you have a partial-write gap.

For Kafka-to-DB pipelines, the correct pattern is still consumer-side dedup with an idempotent DB write (ON CONFLICT DO NOTHING with the Kafka offset as part of the primary key). The 20–30% Kafka transaction cost is then replaced by one cheap DB unique-constraint check. Most production stream-to-DB pipelines choose this hybrid: at-least-once Kafka delivery + idempotent DB consumer.