Idempotency and retries: code and query reading
Read real idempotency, retry, and inbox snippets, predict their behaviour under concurrency or failure, and pick the highest-leverage fix.
Idempotency bugs hide in the gap between two statements, in a missing jitter call, and in a retry condition that fires on the wrong status. Read each snippet the way you would in review — predict the failure under concurrency or load, then pick the fix a senior engineer makes first.
Practise the review loop for this unit: spot the race window, the herd-amplifying retry, the lost transition, and the dedup that silently does nothing — then name the one change that closes each.
Snippet 1 — the dedup handler
func handleCharge(ctx context.Context, db *sql.DB, key string, body []byte) (*Resp, error) {
var existing Row
err := db.QueryRowContext(ctx,
`SELECT response_body, status FROM idempotency_keys WHERE key = $1`, key,
).Scan(&existing.Body, &existing.Status)
if err == sql.ErrNoRows {
// no row yet -> treat as new
db.ExecContext(ctx,
`INSERT INTO idempotency_keys (key, fingerprint, status) VALUES ($1, $2, 'in_progress')`,
key, fingerprint(body))
return process(ctx, db, key, body) // runs the charge
}
return replay(existing), nil
}Two requests with the same key hit this handler within a millisecond. What goes wrong, and what is the single highest-leverage fix?
Snippet 2 — the retry loop
async function callWithRetry(fn, maxAttempts = 6) {
let delay = 200;
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt === maxAttempts - 1) throw err;
await sleep(delay); // fixed doubling, no randomness
delay = Math.min(30000, delay * 2);
}
}
}This loop retries every thrown error with doubling delay. Identify the two distinct production hazards.
Snippet 3 — the state transition
-- request arrives; row already exists for this key
BEGIN;
SELECT status FROM idempotency_keys WHERE key = $1; -- reads 'in_progress'
-- application sees in_progress, calls the payment provider anyway...
-- provider succeeds, then:
UPDATE idempotency_keys
SET status = 'completed', response_body = $2
WHERE key = $1;
COMMIT;A retry races the original request: the original is still in_progress when this block runs. What is the defect in how this code uses the four-state machine?
Snippet 4 — the inbox consumer
BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING;
-- always applied, regardless of whether the insert above did anything:
UPDATE inventory SET reserved = reserved + $qty WHERE sku = $sku;
COMMIT;The relay re-delivers an event after a crash, so this block runs twice for the same event_id. What actually happens, and what is the fix?
Four review reflexes for this unit: a SELECT-then-INSERT idempotency check is a race — make creation atomic with ON CONFLICT DO NOTHING RETURNING and process only the winner; a retry loop without jitter and without a retry-condition is a herd that hammers permanent errors — use full jitter and retry only 5xx/timeouts; an in_progress read must short-circuit to 409, never call the provider again; and an inbox dedup that does not gate the business write on the dedup insert is decorative — the duplicate still applies the effect. Each fix is one or two lines, and each missing line is a duplicate side effect in production.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.