Queues, Streams, Eventing
Delivery guarantees: multiple-choice review
Crux Multiple-choice synthesis across the delivery-guarantees unit — the impossibility proof, the three failure legs, idempotent consumers, Kafka EOS, SQS timeouts, and the outbox.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each mirrors a decision you make in a real incident — not a definition to recite, but a tradeoff to weigh while customers are being double-charged.
Goal
Confirm you can connect the impossibility proof, the failure legs, consumer idempotency, Kafka transactions, SQS mechanics, and the outbox into one coherent end-to-end correctness story.
Quiz
Completed
A vendor's queue advertises 'exactly-once delivery'. A senior reviewer pushes back. What is the precise, correct objection?
Heads-up More replicas improve durability, not two-sided agreement. No amount of replication lets producer and consumer share certainty over a lossy channel; the impossibility is information-theoretic, not a hardware limit.
Heads-up TCP dedups packets within a connection; it says nothing about application-level redelivery after a consumer crash or reconnect. Delivery-guarantee duplicates live above the transport layer.
Heads-up Two-phase commit across producer, broker, and consumer is impractical at queue throughput and still cannot defeat the Two Generals asymmetry. The production answer is one-sided consumer dedup, not distributed agreement.
Quiz
Completed
Across the three failure legs, which one produces the duplicates you actually fight in production, and why does fixing it land on the consumer?
Heads-up Leg 1 lost-ack retries do cause duplicates, but Kafka's idempotent producer neutralises them at ~3% cost. The recurring production pain is Leg 3, where processing already had a visible side effect before the crash.
Heads-up Leg 2 is a loss risk under misconfiguration (acks not all), not a duplicate source. Correctly configured durability does not generate duplicates.
Heads-up A high timeout reduces concurrent-duplicate windows but never eliminates the post-processing-pre-ack crash. At-least-once means duplicates are a structural certainty you design around, not tune away.
Quiz
Completed
A team implements consumer dedup as: SELECT the msg_id; if absent, call Stripe and INSERT it. Under a consumer-group rebalance they still see double charges. What is the structural defect?
Heads-up A longer timeout shrinks the concurrent window but the SELECT/INSERT race is a correctness bug independent of timing — a rebalance or any concurrent delivery re-opens it. The pattern must be made atomic, not just slower to race.
Heads-up The TTL is not the issue here; queues redeliver within seconds. The double charge comes from the non-atomic local dedup letting two side-effect calls slip through before either INSERT lands.
Heads-up An index speeds the lookup but does not serialise two concurrent transactions; only a UNIQUE constraint hit inside the same transaction as the side effect prevents the duplicate.
Quiz
Completed
A Kafka Streams job has enable.idempotence=true and transactional.id set, writing to an output topic AND a Postgres table. Ops still see duplicate Postgres rows after a broker restart. Why?
Heads-up A non-unique transactional.id causes fencing problems, but even with a perfect config Kafka transactions never span Postgres. The duplicate is structural: the DB write is outside the Kafka transaction scope.
Heads-up read_committed only hides aborted Kafka records from the consumer; it has no effect on a write your code makes to Postgres. The DB still needs its own dedup.
Heads-up They already have transactions. No Kafka layer reaches into Postgres — adding more Kafka EOS cost cannot fix a cross-system write. Consumer-side DB dedup does.
Quiz
Completed
An SQS consumer's average processing is 30s; the queue default visibility timeout is 30s. Engineers report sporadic concurrent double-processing. First fix, and what does it buy?
Heads-up maxReceiveCount=1 sends every transient failure to the DLQ on first miss; it does nothing about the timeout-too-short window and turns normal retries into poison-pill investigations.
Heads-up SQS standard queues are at-least-once and there is no 'exactly-once delivery' toggle that defeats Two Generals. The correctness fix is timeout sizing plus an idempotent consumer.
Heads-up The extended value is scoped to that receipt handle; on redelivery the timeout resets to the queue default. You must raise the queue default itself or re-heartbeat every delivery.
Quiz
Completed
A service updates Postgres and publishes a Kafka event as two separate steps. Under load, downstream consumers occasionally never see an event whose DB row clearly committed. Which pattern fixes the root cause?
Heads-up Idempotent producer prevents duplicate publishes; it cannot publish an event that was never sent because the process died after the DB commit. The loss is the dual-write gap, which only a transactional outbox (or CDC) closes.
Heads-up XA/2PC across Postgres and Kafka is brittle, slow, and rarely supported in practice. The outbox achieves the same atomicity using only the DB's local transaction plus an async sender.
Heads-up This is a Kafka producer-side dual-write, not an SQS consumer timing issue. Visibility timeout is irrelevant to a missing publish on the producer side.
Recap
The through-line of the unit is one chain: exactly-once delivery is impossible (Two Generals), so you run at-least-once and move correctness to the consumer. Duplicates concentrate at Leg 3; the cheapest defence is an INSERT-first transactional dedup, extended by a Stripe-style Idempotency-Key for external calls. Kafka’s idempotent producer (~3%) and transactions (~20–30%) only harden the within-Kafka path — cross-system writes still need consumer dedup. SQS visibility timeout must be ~6x processing (or heartbeated), and the outbox pattern closes the producer-side dual-write gap. Broker guarantees are necessary but never sufficient.