Read real consumer code, broker config, and an outbox handler, then pick the answer a senior would commit to: an ack-order bug, a visibility-timeout duplicate, an inline-publish dual write, and an unbounded buffer.
SDSenior◷ 14 min
Level
FoundationsJuniorMiddleSenior
Messaging bugs live in the code and config, not the prose: where you put the ack, what the visibility timeout is set to, whether the relay owns publishing, and whether the buffer has a bound. Read each snippet, trace the crash, and pick the answer a senior engineer would commit to.
Practise the loop you run during a design or code review: locate the load-bearing line, trace what a crash does to it, and choose the change that respects the delivery semantics instead of hiding them.
Snippet 1 — the ack order
def handle(msg): queue.ack(msg) # acknowledged first charge_card(msg.amount) # then the side effect
Quiz
Completed
What delivery semantic does this give, and what's the risk for a payment handler?
Heads-up Acking doesn't run the charge; it tells the broker to forget the message. A crash after ack but before charge_card loses the work with no redelivery. Ack-first is at-most-once (lossy), the opposite of what's claimed.
Heads-up There's no exactly-once over an unreliable network. Ack-first is at-most-once (possible loss); to approximate once-effect you ack AFTER processing (at-least-once) and dedup. The ack's position decides the guarantee.
Heads-up Freeing the queue before the charge runs is exactly how you silently lose a payment on crash. Ack after the charge (at-least-once) and dedup on an idempotency key so the redelivery is harmless.
Snippet 2 — the visibility timeout
# SQS consumervisibility_timeout: 30 # seconds# measured job processing time: p50 = 12s, p99 = 75s# handler is NOT idempotent
Quiz
Completed
With these numbers, what happens to the slowest jobs, and what's the fix?
Heads-up The broker only knows the timeout expired without a delete; it can't tell a slow worker from a crashed one. At 30s it redelivers, so a 75s job runs twice in parallel. Size the timeout above p99 and heartbeat.
Heads-up The DLQ is for messages that FAIL repeatedly, not slow ones. A slow-but-successful job just gets redelivered (duplicated) when the visibility timeout expires. Raise the timeout / heartbeat.
Heads-up That makes it far worse — now even p50 jobs (12s) get duplicated. The timeout must be ABOVE real processing time (p99), not below it, plus heartbeating and idempotency.
Snippet 3 — the outbox handler
with db.transaction() as tx: tx.insert(order) tx.insert(outbox_row(OrderPlaced)) # atomic with the order — good broker.publish(OrderPlaced) # also published inline, "to save a hop" tx.commit()# a separate relay ALSO reads the outbox and publishes
Quiz
Completed
The outbox row is written correctly, so why is the inline `broker.publish` a bug?
Heads-up It quietly recreates the exact problem the outbox removes. A crash after the inline publish but before commit emits an event for an order that rolled back (phantom), and the relay double-publishes. Remove the inline publish.
Heads-up Backwards: the outbox row is the correct, atomic part. The inline publish is the dual-write bug. Keep the outbox + relay; delete the inline call.
Heads-up Retrying a non-transactional publish doesn't make it atomic with the commit; it still risks a phantom event and still duplicates the relay. The fix is structural: only the relay publishes.
Snippet 4 — the buffer
const queue = [] // in-memory, no max sizefunction enqueue(job) { queue.push(job) } // never rejects// worker drains at ~1k/s; producers can burst to ~8k/s during spikes// symptom: during spikes, pod RSS climbs until it's OOM-killed
Quiz
Completed
Why does the pod crash under spikes, and what's the correct change?
Heads-up More memory only delays the OOM under sustained overload — an unbounded queue grows without limit. The fix is a bound that applies backpressure, not a bigger heap that fills more slowly.
Heads-up Faster draining helps, but with no bound a big enough spike still balloons the queue to OOM. You need a bound + fail-fast so producers feel backpressure during the spike, not just more consumers.
Heads-up There's no size that 'always fits' a sustained arrival > service — the queue grows without limit by definition. A bound (and shedding when full) is what keeps the failure small, not a larger buffer.
Recall before you leave
01
How does the ack's position decide the delivery semantic, and what's right for a payment?
02
Why does the inline publish in an outbox handler reintroduce the dual write?
Recap
Every messaging decision in this unit shows up as a load-bearing line you can read straight off the code. The ack’s position sets the delivery semantic — ack-first is at-most-once (lossy), ack-last is at-least-once (duplicates), and a payment wants the latter plus idempotency. The visibility timeout must sit above the real p99 processing time and be heartbeated, or slow jobs get redelivered and run twice in parallel. The outbox only works if the handler does only the local transaction and a separate relay publishes — an inline publish silently reintroduces the dual write (phantom events plus relay duplicates). And an in-memory queue with no bound grows without limit under sustained overload until the process OOMs, so you bound it and fail fast (the backpressure signal), shedding when full and using a durable broker if buffered work must survive a crash. The senior habit is to find the load-bearing line, trace what a crash does to it, and pick the fix the failure mode demands — not the one that hides it.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.