The three failure legs — where duplicates and losses actually happen

The three failure legs

If you have ever wondered exactly where a duplicate slips through — not in theory, but at which network boundary — these are the three places to look. Knowing which leg produced a duplicate tells you which fix to apply.

A message travels across three distinct communication boundaries. Each boundary can fail in ways that produce either loss or duplicates.

Leg 1 — Producer to broker. The producer sends a message and waits for a broker ack. If the network drops the send, the message never arrives. If the network drops the ack, the producer never learns the message was stored — it retries, and the broker receives the message a second time. At-least-once producers retry on timeout; this is a feature, not a bug, but it means the broker can see the same logical message more than once.

Leg 2 — Broker durability. The broker acks the producer but must persist the message before it crashes. If replication lag exists when the broker crashes, followers may not have the message. Proper configuration (Kafka acks=all, SQS durability by default) covers this leg, but misconfiguration silently drops messages here.

Leg 3 — Broker to consumer. This is where most production duplicates happen. The broker delivers a message to the consumer. The consumer processes it (calls Stripe, updates the DB), then sends an ack. If the consumer crashes after processing but before acking, the broker has no record of success — it redelivers the message to the next consumer. The side effect runs twice.

All three legs produce duplicates, but only Leg 3 does so after the side effect has already run — which is why it causes real-world double charges. Legs 1 and 2 produce duplicates or losses before any business logic executes, making them easier to absorb with broker-level config. Without an idempotent consumer, Leg 3 is uncatchable from the broker’s perspective.

Leg 3 is worth tracing one level deeper, because the order of process and ack decides whether a fault becomes a duplicate or a loss. The diagram below follows the consume→process→ack lifecycle and marks where each failure strikes.

Tracing the double-charge incident

Here is the exact SQS visibility-timeout scenario that trips engineers every six months.

1. T=0s: Lambda receives msg-7a3f (“charge $50 for order O-123”). SQS hides the message from other consumers for the visibility timeout (default 30s).
2. T=0.5s: Lambda calls Stripe. Stripe charges the card. charge_id=ch_abc123 returned.
3. T=29s: Lambda is still running (slow downstream DB write). It has not yet called DeleteMessage.
4. T=30s: SQS visibility timeout fires. msg-7a3f becomes visible to all consumers again — it has no idea the first Lambda succeeded.
5. T=30.1s: A second Lambda picks up msg-7a3f. Calls Stripe. Second charge: ch_xyz789.
6. T=31s: First Lambda finally calls DeleteMessage. Message gone. But the damage is done: customer charged $100 for a $50 order.

The failure is not a bug. It is the defined behaviour of at-least-once delivery when visibility timeout is shorter than processing time.