Outbox and inbox: effectively-once across the dual-write boundary

An order service writes the order row to Postgres, then publishes an “order.created” event to Kafka. Between the commit and the publish, the pod crashes. The order exists in the database. The downstream inventory service never sees the event. The order is stuck.

Why dual-write always fails

Any sequence of “write to database, then publish to message broker” has a crash window between the two:

Both sequences are broken. There is no version of “two separate writes” that is atomic.

The outbox pattern: use the database as the broker

Key insight: write the event to the database, inside the same transaction as the business write. The database becomes the single source of truth.

BEGIN;
  -- Business write
  INSERT INTO orders (id, customer_id, total) VALUES ($1, $2, $3);

  -- Outbox write (same transaction)
  INSERT INTO outbox (id, event_type, payload, published)
  VALUES (gen_random_uuid(), 'order.created', $payload, false);
COMMIT;
-- Kafka publish happens AFTER the transaction commits

A separate outbox-relay process polls the outbox table (or tails the Write-Ahead Log via Debezium) and publishes unpublished rows to Kafka, marking them published on broker ACK.

Crash scenarios are now safe:

• Crash before COMMIT → transaction rolls back. Neither the order nor the outbox row exists. No phantom event.
• Crash after COMMIT, before relay publishes → order exists, outbox row published=false. Relay picks it up on next poll.
• Kafka is down → relay retries. Outbox accumulates. Order is safe in DB.

Relay implementations

Poll-based relay: queries WHERE published = false ORDER BY id LIMIT 1000 every few seconds. Simple, adds latency equal to poll interval (typically 1–5 s).

Debezium (CDC relay): tails the Postgres Write-Ahead Log via logical replication. Publishes as soon as the row commits. Sub-second lag. No polling overhead. Production standard for high-throughput services.

AWS SAM variant: DynamoDB Streams → Lambda → EventBridge — managed CDC without running a relay process.

The inbox pattern: deduplicate on the consumer

The relay delivers at-least-once (it may publish twice if it restarts after publishing but before marking the row). The consumer must be idempotent.

Inbox pattern: before applying the event’s business effect, write the event’s id into a processed_events table inside the same transaction as the business write.

BEGIN;
  -- Check if already processed
  INSERT INTO processed_events (event_id) VALUES ($1)
    ON CONFLICT (event_id) DO NOTHING;

  -- If 0 rows affected, this event was already processed — skip
  IF found THEN
    UPDATE inventory SET reserved = reserved - $qty WHERE sku = $sku;
  END IF;
COMMIT;

If the event arrives twice, the second attempt hits the UNIQUE constraint on event_id and the business write is skipped. Effectively-once behavior from the consumer’s perspective.

Inbox table maintenance: partition by event_timestamp and drop partitions older than the broker’s retention window (7–14 days). Without cleanup, the inbox grows forever.

Dead-letter queues: handling unrecoverable failures

Not every failure is recoverable by retry. Malformed data, schema breaks, or violated business invariants are poison messages — no amount of retry will fix them.

After N processing attempts (typically 3–5), move the message to a dead-letter queue (DLQ). The main pipeline keeps flowing; humans review the DLQ.

Main queue → [N retry attempts] → DLQ → human review
                                       → manual replay (after fix)
                                       → formal rejection (business rule)

Production settings: N = 3–5 attempts, DLQ retention 7–30 days, alert on DLQ depth growth. A growing DLQ is a compliance liability — every entry is a record of an operation that may have partial effects.