Backend Architecture
Putting it together: code and incident reading
Crux Read real backend snippets — a blocked event loop, an unbounded pool acquire, a non-atomic idempotency check, and a breaker with no timeout — and pick the fix a senior would make first.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitComposed failures are diagnosed in code and logs, not in the abstract. Read each snippet, predict how it behaves under load, and choose the highest-leverage fix — the one that closes the interaction, not just the local symptom.
Goal
Practise the loop you run in every incident: read the hot path, spot which mechanism’s bad behaviour will feed the next, and reach for the fix that stops the cascade at its source.
Snippet 1 — the handler that blocks the loop
// Node.js, single event loop, thousands of concurrent requests
app.post("/report", (req, res) => {
const body = req.body; // already parsed
const hash = crypto.pbkdf2Sync( // SYNCHRONOUS, ~80ms
body.token, body.salt, 200000, 64, "sha512"
);
const pdf = renderPdfSync(body.rows); // SYNCHRONOUS, ~120ms CPU
res.send({ ok: true, digest: hash.toString("hex"), pdf });
}); Quiz
Completed
Under concurrent load this handler tanks p99 for every endpoint, not just /report. Why, and what is the highest-leverage fix?
Heads-up There is no I/O or pool in this path — it is pure CPU on the event loop. A bigger pool changes nothing; the fix is to stop blocking the loop with synchronous CPU work.
Heads-up Retries on a loop-blocking handler add more CPU work to an already-stalled loop — pure amplification. The fix is to get the synchronous work off the loop, not to repeat it.
Heads-up The cost is not the 200ms of one request — it is that one request's synchronous work freezes the loop and adds latency to thousands of unrelated requests sharing it.
Snippet 2 — the pool acquire with no bound
// Go: a fixed pool of 50 connections, no acquisition timeout
func (h *Handler) Charge(ctx context.Context, req ChargeReq) error {
conn := h.pool.Acquire() // blocks indefinitely if pool is empty
defer h.pool.Release(conn)
// provider call has no timeout of its own either
resp, err := h.provider.Call(conn, req)
if err != nil {
return err
}
return h.repo.Save(conn, resp)
} Quiz
Completed
The provider slows to 4s per call. What does this code do to the whole service, and what is the first fix?
Heads-up Boundedness limits concurrent work but, with no acquire timeout, the 51st caller waits indefinitely. Bounded plus unbounded-wait is how a slow downstream stalls everyone — you must also fail fast on acquire.
Heads-up A bigger pool only delays exhaustion and lets more connections pile onto the slow provider, widening the blast radius. The fix is fail-fast acquisition and a call timeout, not more connections.
Heads-up There is no timeout to make a call fail, so retries never trigger — and if they did, they would grab more connections and amplify the exhaustion. Bound the acquire and the call first.
Snippet 3 — the non-atomic idempotency check
-- Application does: check-then-insert, two separate statements
SELECT result FROM charges WHERE idempotency_key = $1; -- step A: not found
-- ... handler proceeds to charge the card ...
INSERT INTO charges (idempotency_key, result) VALUES ($1, $2); -- step B Quiz
Completed
A client retries the same POST and two requests run concurrently. With this check-then-insert idempotency, what happens and how do you fix it?
Heads-up A separate SELECT then INSERT is not atomic; two concurrent requests both pass the SELECT before either INSERTs, so both charge. Idempotency must be enforced atomically at the storage layer, not by a prior read.
Heads-up Retrying the INSERT does not address the race that already let two charges through. You need a uniqueness constraint so the duplicate write is rejected atomically, not a retry.
Heads-up A plain transaction at the default isolation level still allows both to read 'not found' and both to insert. You need a UNIQUE constraint (or SELECT ... FOR UPDATE / serializable isolation), not just a transaction boundary.
Snippet 4 — the breaker that never trips
const breaker = new CircuitBreaker(callProvider, {
errorThresholdPercentage: 50, // open at 50% errors
resetTimeout: 30000, // half-open after 30s
// no `timeout` option set — calls can hang indefinitely
});
async function charge(req) {
return breaker.fire(req); // awaits callProvider with no deadline
} Quiz
Completed
The provider stops returning — calls hang for minutes rather than erroring. Why does this breaker fail to protect the service, and what is the fix?
Heads-up A lower threshold does not help when there are almost no completed errors to count — the calls are hanging, not erroring. The breaker needs a timeout so hangs become countable failures.
Heads-up resetTimeout governs how fast an open breaker probes recovery; it is irrelevant while the breaker never opens. The missing piece is a call timeout that turns hangs into trip-worthy failures.
Heads-up A call hanging for minutes is the worst kind of failure — it holds resources and never resolves. A breaker without a timeout is blind to exactly the slow failures it most needs to shed.
Recap
Every snippet is one mechanism whose flaw becomes the next mechanism’s catastrophe: synchronous CPU on the event loop stalls all concurrency; an unbounded pool acquire turns a slow downstream into a total stall; a non-atomic idempotency check double-charges under a retry race; and a breaker with no call timeout stays blind to hung calls. The fix is always the one that stops the interaction at its source — move work off the loop, fail fast on acquire, enforce dedup atomically, and give the breaker a timeout — not a knob that defers or amplifies the problem.