Distributed Systems
Leader election: multiple-choice review
Crux Multiple-choice synthesis across the leader-election unit — terms and timeouts, leases vs clocks, split-brain, and the fencing token that the resource must enforce.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each mirrors a call you make in a real incident — not a definition to recite, but a safety argument to defend when a leader pauses, a partition heals, or two nodes both claim the crown.
Goal
Confirm you can connect why we elect a leader, how Raft elects one safely, why a lease is a claim about untrustworthy clocks, and why only a resource-enforced fencing token closes the split-brain window.
Quiz
Completed
A leader holds a 10s lease and writes to object storage; you have observed multi-second GC pauses. Which mitigation actually closes the stale-write window, not just narrows it?
Heads-up A frozen process executes no code, so it cannot 'give up' a lease it does not know expired. A shorter TTL only makes false expirations more frequent and shrinks — never closes — the window.
Heads-up A GC pause can land in the gap between the check and the write — a classic time-of-check/time-of-use hole. Safety cannot depend on the suspect node staying responsive.
Heads-up Election speed is orthogonal. The bug is that the OLD leader still writes after it is deposed; no election speed stops a frozen process from waking and writing stale data.
Quiz
Completed
Why does Raft draw each follower's election timeout randomly from roughly 150–300 ms instead of using one fixed value?
Heads-up Randomization is not about fairness of who leads; it is about staggering candidacy so a majority can form in one round instead of everyone campaigning at once.
Heads-up The range exists to stagger candidacy, not to correct clocks. Raft's safety does not rely on synchronized clocks at all.
Heads-up A fixed timeout breaks no quorum math; it just causes synchronized candidacies that split the vote and force endless retries.
Quiz
Completed
A senior says 'a lease does not bound how long a paused leader keeps believing it leads.' What is the precise reason, and the consequence?
Heads-up NTP reduces skew but never removes the deeper problem: a paused or swapped-out process is not executing, so no clock accuracy helps it notice expiry mid-pause.
Heads-up TTL length changes how often expirations happen, not the fact that a stopped process cannot observe its own expiry. The two-clock gap survives any TTL.
Heads-up The coordinator can revoke its own side, but it cannot reach into a frozen process and stop its in-flight write. The guarantee is one-sided.
Quiz
Completed
Your team adds fencing tokens: the lock service hands out strictly increasing numbers and the leader stamps every write with its token. Incidents continue. What is the most likely gap?
Heads-up Strict monotonicity is all that matters, not the size of the increment. A token of 34 beats 33 no matter the gap; the failure is the resource not checking.
Heads-up The stale leader is paused and bypasses the lock service entirely when it wakes and writes directly to storage. Only the resource sees that write, so the resource must enforce.
Heads-up Tokens are compared as integers, not timestamps. The whole point of fencing is to be correct WITHOUT trusting clocks.
Quiz
Completed
A 5-node cluster splits 3-2 by a network partition. How does a correct quorum-based election protocol behave, and why?
Heads-up That IS split-brain with data loss. A majority-quorum protocol forbids it precisely so writes never diverge — the minority side is barred from electing.
Heads-up A leader needs a MAJORITY (3 of 5), not all nodes. The majority side stays available; tolerating one minority partition is the entire point.
Heads-up If the old leader is stranded in the minority it loses the ability to commit because it cannot reach a quorum. Leadership follows the quorum, not history.
Quiz
Completed
You need 'run this cron on exactly one host' and reach for a distributed lock. A reviewer warns the lock alone is not enough for correctness. What is the missing piece?
Heads-up More lock replicas improve availability, not safety. They do nothing about a holder that pauses and wakes to write after its lock expired.
Heads-up Idempotency helps for some workloads, but it is a different property; many side effects (sending money, appending to a file) are not naturally idempotent, and the question is about lock correctness.
Heads-up The job's external writes (object storage, another service) are outside that transaction, so the transaction cannot fence them. The token check must live at each protected resource.
Recap
The through-line of the unit is one decision tree: elect a single leader to serialize writes (Raft uses terms plus randomized 150–300 ms timeouts to elect one safely), hold leadership as a lease that the coordinator can expire — but never trust that a paused leader knows it lost the lease. A partition is handled by quorum (only the majority leads); a pause is handled by a monotonic fencing token that the protected resource itself enforces. Elect for liveness, fence for safety — and remember a bare lock or lease without resource-side token checking buys you nothing against a stale write.