Backend Architecture
Idempotency and retries: multiple-choice review
Crux Multiple-choice synthesis across the idempotency + retries unit — keys, the four-state machine, jittered backoff, outbox/inbox, concurrency races, and production failure modes.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. None is a definition to recite — each is a decision you make under load, where the wrong call is a double charge, a lost event, or a self-inflicted outage.
Goal
Confirm you can connect the moving parts the lessons built toward: the idempotency key, the server-side state machine, jittered retries, the outbox + inbox composition, and the concurrency races that turn a tidy design into a duplicate side effect.
Quiz
Completed
A mobile client fires a POST /charge, the response is lost, and it retries. On the retry it generates a fresh UUID for the Idempotency-Key header. What is the consequence and why?
Heads-up The fingerprint is keyed off the idempotency key. With no matching key the server has no row to compare against, so the body hash never gets consulted — it processes a new charge.
Heads-up 422 (mismatch) requires an existing row for that key with a different fingerprint. A brand-new key has no row, so it is treated as new (200/201), not 422.
Heads-up There is no automatic reconciliation. The whole point of the key is to make the server dedup; a new key per attempt removes that mechanism.
Quiz
Completed
A request arrives with Idempotency-Key K1. The server finds a row for K1 with status=in_progress. Why return 409 immediately instead of blocking until the first request finishes?
Heads-up RFC 9110 says nothing about idempotency-key concurrency; 409 here is Stripe's widely-adopted convention, chosen for connection efficiency, not spec compliance.
Heads-up The row is readable at any time — that is exactly how the server knows to return 409. The choice is about not holding a connection, not about read access.
Heads-up The client must reuse the same key on retry. 409 signals 'already in flight, back off' — regenerating the key would create a second charge.
Quiz
Completed
A service returns 503 with Retry-After: 30 during an outage. Across the fleet, which client behaviour keeps the recovering service alive?
Heads-up A synchronised 1s retry from every client is the thundering herd — the same spike that broke the service. Ignoring Retry-After is the textbook anti-pattern.
Heads-up Without jitter all clients that failed at the same instant pick the identical delay and retry in the same tick. The doubling does not desynchronise them.
Heads-up A 6-attempt cap per request still lets 10k concurrent callers generate 60k retries in a short window. The budget is a fleet-wide ceiling the per-request cap cannot provide.
Quiz
Completed
A team must publish an 'order.created' event to Kafka whenever an order row commits to Postgres. Why is the outbox pattern correct where 'write Postgres then publish Kafka' is not?
Heads-up Kafka-then-Postgres just moves the window: a crash after publish but before commit leaves a phantom event for a write that never happened.
Heads-up Async or not, the publish and the commit are two separate operations with a crash window between them. There is no atomic 'two separate writes'.
Heads-up The relay still delivers at-least-once (it may re-publish after crashing before marking a row published). The consumer's inbox dedup is what makes it effectively-once.
Quiz
Completed
Two requests carrying the same idempotency key reach the server within a millisecond of each other. The handler does SELECT-then-INSERT into the idempotency_keys table. What happens, and what is the fix?
Heads-up Plain SELECT-then-INSERT has no lock between the statements; nothing serialises the two requests. The race window is real.
Heads-up If a unique constraint exists you should use ON CONFLICT to turn the collision into a controlled 409/replay, not crash with 500. The naive code may also have no constraint at all.
Heads-up SERIALIZABLE can abort one transaction, but you still must retry and handle the conflict; the clean, single-statement fix is ON CONFLICT DO NOTHING RETURNING.
Quiz
Completed
A payment API uses Redis alone (async fsync, SETNX) as its idempotency store. A node crashes a millisecond after a SETNX, losing that key. What is the production-grade reading of this design?
Heads-up Default Redis replication is async; a crash before the replica receives the write loses the key. Replication reduces but does not eliminate the window.
Heads-up If the charge already hit the payment processor before the crash, it stands. Losing the dedup key does not undo a side effect that already happened.
Heads-up A legal liability for double charges needs an architectural guarantee, not a low probability. Capping retries lowers the odds but does not remove them.
Recap
The unit is one chain: the client retries with one key per operation; the server resolves that key to new / in-flight (409) / replay / mismatch (422) and stores the result atomically with ON CONFLICT; retries use full jitter, honour Retry-After, and live under a fleet retry budget; the outbox + at-least-once relay + inbox dedup turn the impossible dual-write into effectively-once; and the durable store must survive a crash, because losing a key is how a tidy design becomes a double charge. Every production failure in the unit — Stripe 2017, Knight Capital, AWS S3, GitHub 2018 — is at-least-once without idempotent consumers, or retries without jitter.