End-to-end order pipeline: where each delivery guarantee lives

A customer is charged twice for one order. The trace tells the story: the payment consumer processed the order.placed event, called the gateway, succeeded — and then the pod was killed mid-commit, before the Kafka offset moved. The group rebalanced, a new consumer picked up the same offset, and replayed the event. Two charges, one order. The gateway call had no idempotency key, so it had no way to know it was the same charge. The pipeline was correct everywhere except the one place that mattered.

One write, six hops, one guarantee that ties them

Follow a single order through an event-driven pipeline and you cross six boundaries, each with its own failure mode. The write happens once. Everything after is a relay across networks and processes that can each retry, crash, reorder, or stall. The capstone insight is that you do not get one global “exactly-once” knob. You choose a guarantee at each hop, and then you defend the seams with one invariant — idempotency — because every honest hop in this chain is at-least-once.

The flow:

Hop 1–2: the write is atomic, the publish is not

The first decision is the famous dual-write problem: you must save the order and emit order.placed. If you write to the DB and then publish to Kafka as two separate calls, a crash between them either loses the event (order exists, nobody knows) or emits a phantom (event sent, order rolled back). The transactional outbox removes the gap: in one local DB transaction you write the order row and an outbox row. Either both commit or neither does. The event now exists if and only if the order does.

Publishing is a separate concern, and it is honestly at-least-once. A CDC relay (Debezium tailing the Postgres WAL is the canonical setup) reads committed outbox rows from the log and pushes them to Kafka. Its superpower is recoverability: it tracks its log offset, so on restart it resumes from the exact position — zero polling, near-zero DB load because it reads the log, not the table. But “resume from offset” means it can re-emit a row it published just before crashing. That is fine, and it is the whole point: you are not chasing exactly-once on the wire; you are pushing the dedup responsibility to the consumer, where it belongs.

Hop 3: partition by the key that owns the order

Kafka guarantees ordering only within a partition, and the partition is chosen by the record key. Key your order events by order_id (or customer_id when you need a customer’s events serialized) and every event for that order lands in the same partition, processed in commit order by one consumer. payment.authorized can never overtake order.placed for the same order. Cross-order ordering is neither guaranteed nor needed — that is exactly what lets you scale by adding partitions.

The trap is operational: the default partitioner is hash(key) % partitionCount. The moment you add partitions to scale, the modulus changes, the same key maps to a different partition, and in-flight ordering for that key breaks at the seam. Senior teams over-provision partitions up front, or migrate keys deliberately, rather than bumping partition count on a hot Friday. Partition count is a one-way door for key ordering.

Hop 4–5: at-least-once is a promise to deliver duplicates

The consumer group is where the load-bearing rule lives. To get at-least-once you commit the offset only after the handler succeeds — process, then commit. The cost is duplicates: if the pod dies after processing but before the commit (exactly the Hook), the next consumer replays that offset. At-least-once is not a bug to be fixed; it is a contract that says “I will deliver your message, possibly more than once.” The only correct response is an idempotent handler.

Idempotency means processing the same event twice has the same effect as once. The mechanism is a dedup key — the event id, or a business idempotency key — recorded in a dedup table with a unique constraint, inside the same transaction as the side effect. Re-delivery hits the constraint and is a no-op. For external calls, propagate the key downstream: Stripe’s Idempotency-Key header dedups for 24 hours, so passing your event’s key into the charge call means a retry returns the original charge instead of creating a second one. That single header is the difference between the Hook being a near-miss and a double charge.

Some failures are not transient — a malformed payload, a permanently-rejected card. Retrying those forever blocks the partition (head-of-line blocking) and lets lag grow without bound. So you give each message a retry budget with exponential backoff for transient errors, and on exhaustion route it to a dead-letter queue. Uber’s reliable-reprocessing design uses tiered retry topics feeding a DLQ precisely so a poison message is quarantined instead of stalling the live stream. The DLQ is not a graveyard; it is an inbox you must drain.

Hop 6: the UI must tell the truth about lag

Because every hop is asynchronous, the moment the user clicks “Place order” the answer is not yet known. Inventory and notification may settle hundreds of milliseconds later; under load they settle when consumer lag drains. A senior UI does not lie about this. It applies an optimistic update so the order appears instantly, marks it pending until a confirmation event arrives, and reconciles to confirmed or failed from the real state. The eventual-consistency window is a product surface, not a defect to hide.

This is also where observability earns its keep. The metrics that catch the silent killers are consumer lag (the gap between latest offset and committed offset — Burrow watches its trend over a sliding window, flagging OK / WARNING / ERR rather than a raw number), DLQ depth (a rising DLQ means handlers are failing faster than you drain), and end-to-end latency with a correlation id stamped on the event at hop 1 and carried through every async hop so one trace spans the whole journey. The Hook’s double charge is invisible without that trace. Lag growing until the DLQ overflows is invisible without those two gauges and an alarm on each.