Queues, Streams, Eventing
Queues capstone: multiple-choice review
Six questions that cut across the whole queues track. None is a definition to recite — each one is a decision you make while designing or debugging one order-processing pipeline, where the outbox, Kafka, CDC, the DLQ, and the UI all have to agree.
Confirm you can hold the whole pipeline in your head at once: where each delivery guarantee lives, why per-key ordering is fragile, and why one invariant — idempotency — is what makes every hop safe.
Across the order pipeline — outbox write, CDC publish, Kafka delivery, consumer processing — which single invariant makes the end-to-end system correct, and why?
The order handler must both persist the order and emit order.placed. Why is the transactional outbox the right shape rather than INSERT-then-kafka.publish in the same handler?
The order topic is keyed by order_id with 6 partitions. Lag spikes on a hot Friday and someone bumps it to 12 partitions. What breaks, and why?
The payment consumer pulls order.placed, calls Stripe, succeeds, then the pod is killed before the Kafka offset commits. The group rebalances and replays the offset. How do you prevent a double charge?
You use CDC (Debezium tailing the WAL) as the outbox relay. A Kafka Connect worker OOMs and stays down for hours. What is the production risk, and the guardrail?
The checkout POST returns 202 Accepted in 40ms; a consumer writes the order ~700ms later. The UI shows a success toast and immediately refetches the order list. What does the user experience, and what is the honest design?
The whole track collapses into one design tree: the order write is atomic because the row and its outbox event commit in one local transaction; CDC (or a polling relay) publishes at-least-once by resuming from its log offset — and the slot is a loaded gun you must alert on and cap; the topic preserves per-key order by keying on order_id, which is why bumping partition count is a one-way door that rehashes keys; the consumer group delivers at-least-once by committing offsets only after processing; poison messages that exhaust a retry budget go to the DLQ instead of blocking the partition; and the UI tells the truth about the consistency window with optimistic-then-pending state. The thread through all of it is idempotency — a dedup key under a unique constraint plus an Idempotency-Key propagated to external calls — because every honest hop is at-least-once and duplicates are the contract, not the bug.