Backend Architecture
Idempotency and retries: code and query reading
Crux Read real idempotency, retry, and inbox snippets, predict their behaviour under concurrency or failure, and pick the highest-leverage fix.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitIdempotency bugs hide in the gap between two statements, in a missing jitter call, and in a retry condition that fires on the wrong status. Read each snippet the way you would in review — predict the failure under concurrency or load, then pick the fix a senior engineer makes first.
Goal
Practise the review loop for this unit: spot the race window, the herd-amplifying retry, the lost transition, and the dedup that silently does nothing — then name the one change that closes each.
Snippet 1 — the dedup handler
func handleCharge(ctx context.Context, db *sql.DB, key string, body []byte) (*Resp, error) {
var existing Row
err := db.QueryRowContext(ctx,
`SELECT response_body, status FROM idempotency_keys WHERE key = $1`, key,
).Scan(&existing.Body, &existing.Status)
if err == sql.ErrNoRows {
// no row yet -> treat as new
db.ExecContext(ctx,
`INSERT INTO idempotency_keys (key, fingerprint, status) VALUES ($1, $2, 'in_progress')`,
key, fingerprint(body))
return process(ctx, db, key, body) // runs the charge
}
return replay(existing), nil
} Quiz
Completed
Two requests with the same key hit this handler within a millisecond. What goes wrong, and what is the single highest-leverage fix?
Heads-up A timeout is good hygiene but unrelated to the double charge. The defect is the non-atomic SELECT-then-INSERT race window between the two requests.
Heads-up The code ignores the INSERT's error and unconditionally calls process(). Even if the PK rejects the second INSERT, both goroutines already passed the ErrNoRows branch and both charge.
Heads-up fingerprint is microseconds and irrelevant to the race. The race is the gap between reading no row and inserting one — closed only by an atomic insert-or-noop.
Snippet 2 — the retry loop
async function callWithRetry(fn, maxAttempts = 6) {
let delay = 200;
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt === maxAttempts - 1) throw err;
await sleep(delay); // fixed doubling, no randomness
delay = Math.min(30000, delay * 2);
}
}
} Quiz
Completed
This loop retries every thrown error with doubling delay. Identify the two distinct production hazards.
Heads-up Those values are within normal ranges (base 100–500ms, cap 30–60s). The hazards are the missing jitter and the unconditional retry of non-retryable errors, not the constants.
Heads-up Six attempts does not exhaust memory. The real scale problem is fleet-wide — without a retry budget many callers each doing 6 attempts amplify load — but the snippet's local bugs are no jitter and retrying 4xx.
Heads-up await sleep yields; it does not block the event loop. The defects are the synchronised (un-jittered) delay and retrying errors that will never succeed.
Snippet 3 — the state transition
-- request arrives; row already exists for this key
BEGIN;
SELECT status FROM idempotency_keys WHERE key = $1; -- reads 'in_progress'
-- application sees in_progress, calls the payment provider anyway...
-- provider succeeds, then:
UPDATE idempotency_keys
SET status = 'completed', response_body = $2
WHERE key = $1;
COMMIT; Quiz
Completed
A retry races the original request: the original is still in_progress when this block runs. What is the defect in how this code uses the four-state machine?
Heads-up The row already exists (status=in_progress), so INSERT is wrong. The defect is processing at all on an in_progress read instead of returning 409.
Heads-up Isolation does not fix the logic error: the code deliberately calls the provider after seeing in_progress. The state machine itself requires 409 on in_progress, regardless of isolation.
Heads-up The external charge already happened at the provider and cannot be rolled back by the DB. The fix is to never call the provider on an in_progress read.
Snippet 4 — the inbox consumer
BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING;
-- always applied, regardless of whether the insert above did anything:
UPDATE inventory SET reserved = reserved + $qty WHERE sku = $sku;
COMMIT; Quiz
Completed
The relay re-delivers an event after a crash, so this block runs twice for the same event_id. What actually happens, and what is the fix?
Heads-up ON CONFLICT DO NOTHING suppresses the error — there is no throw. The block commits and the unconditional UPDATE double-reserves.
Heads-up The inbox table only records the event_id; it does nothing to the UPDATE unless the code branches on whether the insert succeeded. Here it does not, so the UPDATE always runs.
Heads-up The +/- direction is domain-specific and not the defect. The defect is the business write running on a duplicate delivery because it is not gated on the dedup insert.
Recap
Four review reflexes for this unit: a SELECT-then-INSERT idempotency check is a race — make creation atomic with ON CONFLICT DO NOTHING RETURNING and process only the winner; a retry loop without jitter and without a retry-condition is a herd that hammers permanent errors — use full jitter and retry only 5xx/timeouts; an in_progress read must short-circuit to 409, never call the provider again; and an inbox dedup that does not gate the business write on the dedup insert is decorative — the duplicate still applies the effect. Each fix is one or two lines, and each missing line is a duplicate side effect in production.