Distributed Systems
Distributed capstone: multiple-choice review
Crux Cross-track multiple-choice synthesis — how quorum, leader election, clocks, sagas, and retries compose into one pipeline, and where the seams leak.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole track. Each one is a decision an on-call engineer makes when a pipeline of individually-correct services still produces a wrong number on a customer’s statement. None of them is a definition to recite.
Goal
Confirm you can connect quorum replication, leader election with fencing, logical clocks, saga compensations, and retry discipline into one system — and locate the seam where two correct layers compose into a bug.
Quiz
Completed
An order pipeline of four services double-refunds a customer occasionally. Every service passes its own unit tests; the payment service is idempotent for charges; the saga is textbook; retries have backoff and jitter. Where does the bug live?
Heads-up The refund path passes its tests and the service is idempotent for charges. Composition failures are exactly the ones no single component's tests can catch, because each layer is correct in isolation — the bug is in the seam, not the layer.
Heads-up Brokers deliver at-least-once by design; duplicates are expected, not a broker bug. The pipeline must be safe under duplicates via idempotency keys — it must never assume the broker prevents them.
Heads-up R + W > N is the property you want — it gives read-your-write overlap and durability. It governs replica visibility, not whether a retried compensation double-fires.
Quiz
Completed
The saga orchestrator runs as an elected leader. It suffers a long GC pause, its lease expires, a new leader is elected — then the old one wakes and tries to write a saga step. What actually prevents it from corrupting state the new leader already advanced?
Heads-up It cannot reliably know — that is the whole problem. A paused process believes no time passed and may think it still holds the lease. Only the resource enforcing a fencing token is safe against a stale, paused leader.
Heads-up A lease bounds wall-clock time, but a paused process has no idea time elapsed. Kleppmann's point: without a fencing token a stale leader can wake after its lease expired and still issue a write.
Heads-up Backoff spaces out retries; it does nothing to reject a write from a node that should no longer be writing. Rejecting the stale write is the fencing token's job, not the retry policy's.
Quiz
Completed
A teammate proposes ordering saga steps across services by comparing wall-clock timestamps (Date.now()) so that 'Payment before Shipping' is enforced. Why does a senior reject this, and what is the correct mechanism?
Heads-up Performance is not the issue. The problem is correctness: physical clocks across hosts are not synchronized tightly enough to establish causal ordering, regardless of how fast the call is.
Heads-up NTP bounds skew to tens of milliseconds at best and can step the clock backwards. Cross-service event ordering at sub-second granularity cannot rely on it; you need logical ordering.
Heads-up Same datacenter narrows skew but never eliminates it, and clock steps still happen. Co-location reduces — but does not remove — the need for a logical or bounded-uncertainty clock.
Quiz
Completed
Why is idempotency called the load-bearing primitive of the pipeline, rather than just one safeguard among many?
Heads-up Idempotency does not remove retries; it makes them safe. A dropped request still needs a retry to complete. Idempotency guarantees the retry has no extra effect, not that no retry happens.
Heads-up Exactly-once delivery over a network is impossible. What is called 'exactly-once' is at-least-once delivery plus idempotent processing keyed on a dedup id — the network gives duplicates, the key gives exactly-once effects.
Heads-up 'decrement stock by 1' is naturally repeatable and still wrong when repeated. Idempotency is not natural repeatability — it is a stable business key the receiver records so a second arrival is a no-op returning the first result.
Quiz
Completed
A retry storm is amplifying a small downstream fault into a full outage: every layer retries the layer below it. What is the structural fix, and what number anchors it?
Heads-up Longer timeouts hold connections open longer and can worsen a storm by piling up in-flight work. They do not bound the multiplication of load; only a budget caps how much retries can add.
Heads-up No backoff is exactly what turns a transient fault into a thundering herd — synchronized immediate retries amplify load. Backoff plus jitter spreads retries; the budget bounds their total.
Heads-up Scaling capacity delays the collapse but the storm scales with traffic and re-saturates the new capacity. The durable fix bounds retry amplification at the source with a budget, not by absorbing it.
Quiz
Completed
Every service dashboard is green, yet the pipeline is silently failing customers. Which signal set does a senior actually watch to catch composition failures?
Heads-up Those are exactly the green dashboards that hide composition failures. Each component is fine in isolation; the failure lives between components, which per-service node metrics never surface.
Heads-up Edge error rate tells you something is wrong but not where, and it lags the cause. Seam signals (consumer lag, leader churn, retry-budget) let you catch and locate the failure before it reaches the edge.
Heads-up A double refund or a stalled saga does not move aggregate throughput. Throughput is blind to correctness-at-the-seam failures; you must alert on the inter-layer signals directly.
Recap
The track’s through-line is one decision tree: quorum makes a write durable, a leader with fencing tokens makes coordination single-writer-safe, logical clocks order steps causally, sagas reverse with compensations because there is no distributed ACID, and retries need a budget so a fault cannot amplify. Idempotency keyed on business intent is the load-bearing primitive that makes at-least-once safe everywhere. The real failures live in the seams — a retry re-firing a compensation with no shared key — and you catch them on seam signals, not on green per-service dashboards.