Leader election
Some jobs need exactly one coordinator. Leader election picks it — via a consensus protocol (Raft/ZAB) or a lease/lock in ZooKeeper, etcd, or Redis. The deep trap is split-brain: a paused leader that thinks it's still in charge. The fix is a fencing token, not a longer timeout.
A billing system ran a nightly job to charge subscriptions. To survive a crash, the team ran the scheduler on three machines and used a distributed lock so only the holder would run the job. One night, the lock-holder’s JVM hit a 25-second garbage-collection pause right after acquiring the lock. The lock’s lease expired, a second machine grabbed it and started charging customers — and then the first machine woke up, still believing it held the lock, and charged everyone again. Two coordinators, one job, double charges. The bug was not the lock; it was assuming “I acquired the lock” stays true forever. Electing a single coordinator is easy. Guaranteeing there is never more than one — even when a node freezes — is the hard part.
Why you sometimes need exactly one
Most of system design pushes toward statelessness and “any node can do anything” — it scales and tolerates failure. But some tasks are inherently singular: they must be done by exactly one node at a time, or they break. A scheduler that must fire each cron job once (not three times); the single writer/primary in a replicated database; the one node assigning partitions or sequence ranges; the coordinator that compacts a shared log. Run two of these at once and you get duplicate side effects, conflicting writes, or corrupted state.
Leader election is the mechanism that designates one node — the leader — to do the singular work, while the others stand by ready to take over if it dies. The two hard requirements are uniqueness (never two leaders simultaneously) and liveness (when the leader dies, a new one is elected promptly). Uniqueness is the one people underestimate, and the one that causes the worst outages.
Two ways to elect: consensus vs a lease
Before choosing a coordination service, ask yourself one question: is a duplicate operation merely wasteful, or is it catastrophic? That single question determines how much correctness machinery you actually need — and it’s the one engineers routinely skip, which is how double-charges happen.
There are two families of mechanism, and most real systems use the second built on top of the first.
Consensus protocols (Raft, ZAB, Paxos). A cluster of nodes runs a protocol where a majority (quorum) must agree on who the leader is. In Raft, nodes vote in numbered terms; a candidate that wins votes from a majority becomes leader for that term, and because a majority is required, two leaders can’t be elected in the same term — any two majorities overlap in at least one node, which won’t vote twice. The leader sends heartbeats; if followers stop hearing them, they start a new term and elect again. This is how etcd, Consul, and Kafka’s controller (KRaft) elect, and how a replicated state machine stays consistent.
A lease or lock in a coordination service. Rather than run consensus yourself, you lean on a service that already does — ZooKeeper, etcd, or a lock in Redis — and treat “holding the lock/lease” as “being the leader.” A node acquires a time-bounded lease; while it holds the lease it’s the leader; it must renew before expiry or it loses leadership, and another node can acquire it. This is simpler to build (the hard consensus lives in the proven service) and is the common pattern for “I just need one worker to run this job.”
Split-brain: the failure that defines the problem
The reason leader election is hard is split-brain: a situation where two nodes both believe they are the leader at the same time. It happens not because the election code is buggy, but because of the gap between being the leader and knowing you still are:
- A network partition isolates the leader from the cluster. The others can’t hear its heartbeats, declare it dead, and elect a new leader — but the old leader, on its side of the partition, still thinks it’s in charge.
- A GC pause, VM suspension, or long stall freezes the leader past its lease expiry. The cluster elects a new leader. The frozen node wakes up still holding its now-expired belief and acts as leader — exactly the billing-system hook.
The naïve “fix” — make the lease longer — only widens the window in which a real failure goes unhandled, trading one problem for another. A long timeout makes split-brain rarer but slower failover; a short one makes failover fast but split-brain more likely. Tuning the timeout cannot eliminate split-brain, because no timeout can distinguish “dead” from “frozen and about to wake up.” You need a mechanism that makes a stale leader’s actions harmless even if it acts.
▸Why this works
Why does a majority quorum prevent two leaders in consensus, but a lease alone doesn’t? In Raft, becoming leader requires votes from a strict majority of nodes, and each node votes at most once per term. Any two majorities of the same set must share at least one node — and that shared node won’t vote for two different candidates in one term — so at most one leader can win per term. The math forbids two leaders in the same term. A bare lease has no such overlap argument across the whole system: the coordination service grants one lease, but a leader that pauses past expiry and then resumes still thinks it holds it, and if it talks directly to a resource (a database, a file) without re-checking, nothing stops it. That’s why a lease needs a fencing token: it ports the “monotonic, can’t-go-back” property of consensus terms down to the resource being protected.
The fencing token: the real fix
The correct defence against split-brain is a fencing token: a monotonically increasing number the coordination service hands out with each leadership grant. Lease #1 carries token 33; when it expires and a new leader takes over, that grant carries token 34. The crucial rule lives at the protected resource: every operation includes its token, and the resource rejects any token lower than the highest it has already seen.
1. Leader A acquires lease, gets fencing token = 33
2. A sends "write X" with token 33 → resource accepts (33 ≥ 33), remembers 33
3. A freezes (GC pause); lease expires
4. Leader B acquires lease, gets token = 34
5. B sends "write Y" with token 34 → resource accepts (34 ≥ 34), remembers 34
6. A wakes up, still thinks it's leader, sends "write Z" with token 33
→ resource REJECTS: 33 < 34. A's stale write is fenced off. ✓Now even if the zombie leader resumes and tries to act, its operations carry an old token and are refused — the harm is structurally impossible, regardless of timeouts. This is why “use a longer lock timeout” is the wrong answer and “the resource must enforce a fencing token” is the right one: correctness comes from the resource rejecting stale tokens, not from hoping the old leader stays asleep.
▸Common mistake
Treating a Redis lock as a correctness guarantee without fencing. A single-node Redis SET key value NX PX <ttl> lock is convenient and great for efficiency (avoid two workers doing the same work — a duplicate is wasteful but not catastrophic). It is not safe for correctness (where a duplicate causes double-charging or data corruption), because it has no fencing token and its TTL can expire under a GC pause exactly as in the hook. If two workers running the job is merely wasteful, a plain lock is fine. If two workers is a disaster, you need fencing tokens enforced at the resource — and a lock service whose tokens are linearizable (etcd/ZooKeeper). Decide which case you’re in before choosing the lock, not after the incident.
A leader acquires a lock, then suffers a 30-second GC pause. Its lease expires, a new leader is elected, and the old one wakes up still believing it's the leader. Both now act. What is this called, and what actually prevents the damage?
In Raft, why can't two nodes both be elected leader within the same term?
To make split-brain harmless, the coordination service issues a monotonically increasing _______ with each leadership grant, and the protected resource rejects any operation carrying a lower one — so a revived stale leader's writes are refused no matter what the timeouts were.
- 01When do you need leader election, and what are its two requirements?
- 02Contrast consensus-based election with a lease/lock, and why a majority prevents two leaders.
- 03What is split-brain, why doesn't a longer timeout fix it, and what does?
Leader election designates exactly one node to do inherently singular work — a once-only scheduler, the single write-primary, a partition assigner — where running two leaders causes duplicate side effects or corruption. Its two requirements are uniqueness and liveness, and uniqueness is the dangerous one. You elect either via a consensus protocol (Raft/ZAB/Paxos), where a majority quorum mathematically forbids two leaders in the same term because any two majorities share a voter, or via a renewable lease/lock in a coordination service (ZooKeeper, etcd, Redis) that already runs consensus for you. The failure that defines the whole problem is split-brain — two leaders at once, caused by a network partition or a GC pause/VM stall that outlasts the lease, after which a frozen node wakes up still acting as leader (the billing double-charge). Crucially, a longer timeout cannot fix this — no timeout tells “dead” from “frozen.” The real fix is a fencing token: a monotonically increasing number issued per leadership grant that the protected resource enforces by rejecting any lower token, so a revived stale leader’s writes are structurally refused. Correctness lives at the resource, not in the lease length — which is also why a plain Redis lock is fine for efficiency but unsafe for correctness without fencing. Now when you see an incident where one job fired twice, or a once-per-cluster task ran on two nodes simultaneously — ask first: does the resource enforce a fencing token, or is it trusting the leader to stay alive?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.