Raft roles, terms, and why majority quorums prevent split brain
Raft''''s three node roles, the monotonic term counter, and the quorum rule that makes it impossible for two leaders to exist at once.
Your Kubernetes cluster is backed by a 5-node etcd cluster. One node loses power mid-morning. You run kubectl get pods and it works fine — no error, no stall. How is there still agreement with a node down?
The job: one machine from many
A distributed system’s hardest problem is agreement. If five nodes each accept writes independently, you get five conflicting histories. Raft’s job is to make those five nodes behave like one: same order of changes, same state, no writes lost. It does this by electing exactly one leader at a time and routing all changes through that leader.
Three roles
Understanding the three roles is what lets you read a Raft status page — or a post-mortem — and immediately know which invariant held and which one broke.
Every Raft node is in exactly one of three states:
- Follower — the default state. Receives and stores log entries from the leader. Does not accept client writes directly.
- Candidate — a follower that has stopped hearing from a leader and is now running for leadership. Temporary state, lasts until the election resolves.
- Leader — the node clients send writes to. Drives replication to all followers. At most one leader per term exists in a healthy cluster.
A node starts as a follower. It becomes a candidate when its election timeout fires. It becomes a leader if it wins a majority vote.
The term: a monotonic logical clock
Raft tracks time not with wall clocks but with terms — monotonically increasing integers. Each term begins with an election. If a leader wins, it leads for the whole term. If no leader emerges (split vote), the term ends and a new one starts.
The term has two jobs:
- Deduplication. When a message arrives, nodes compare the sender’s term to their own. A higher term always wins — the receiver updates its term and steps down to follower if needed. This resolves stale-leader confusion instantly.
- Ordering. Every log entry is tagged with the term it was written in. This tag is used later to detect log divergence.
Together, these two jobs mean the term is the single shared clock that makes “who is in charge right now?” always answerable without any wall-clock agreement — drop either job and you get either a zombie leader or an undetectable log split.
| Term | What happened |
|---|---|
| 1 | Node A elected leader. Served 30 s. |
| 2 | A lost network briefly. B won election. |
| 3 | B crashed. C won election. |
| 4 | C still leader — no new election needed. |
Terms are never reused. If you see term 7, every message from term 6 is stale.
Majority quorum: the split-brain barrier
Raft requires a majority (more than half the cluster) for two operations: elections and commits. In a 5-node cluster the majority is 3.
Why majority specifically? The key property is overlap: any two majorities of the same set share at least one node. In a 5-node cluster, if one set of 3 commits an entry and a different set of 3 elects a new leader, those two sets cannot be disjoint — they share a node. Through that shared node, the new leader is guaranteed to have seen the committed entry.
If Raft used simple plurality (2 of 5) instead of majority, two separate groups of 2 could each believe they are authoritative — split brain. Majority prevents this.
Failure tolerance: a cluster of N nodes tolerates floor((N-1)/2) simultaneous failures. 5 nodes → 2 failures. 3 nodes → 1 failure. This is why Raft clusters are 3, 5, or 7 nodes — odd numbers maximize tolerance for a given size.
A 5-node Raft cluster is split: DC A has the leader and 2 followers (3 nodes), DC B has 2 followers. The link between DCs is cut. What happens?
Why does Raft require a majority (3 of 5), not just any 2 of 5, for both elections and commits?
Put the Raft leader-election steps in order:
- 1 Followers stop receiving heartbeats for longer than the election timeout
- 2 A follower transitions to candidate, increments its term, and votes for itself
- 3 The candidate sends RequestVote RPCs to all other nodes
- 4 Each node grants its vote at most once per term, to the first eligible candidate
- 5 The candidate collects a majority of votes and becomes leader for the new term
- 6 The new leader starts sending heartbeats to assert its authority
Fill in the blank: Raft uses a council of N members where only one member at a time holds the _______ and proposes new laws.
- 01Why does a 5-node Raft cluster survive 2 simultaneous failures but not 3?
- 02What is a Raft term and why does it replace wall-clock time?
- 03A node in a Raft cluster has been offline for 10 minutes. It comes back with term 4, but the cluster is now on term 9. What happens when it sends a message?
Raft assigns each node one of three roles — follower, candidate, or leader — and exactly one leader exists per term. The term is a monotonic logical clock that resolves stale-leader confusion: higher term always wins. Both elections and commits require a majority quorum, which guarantees that any two quorums share at least one node — making it impossible for two separate leaders to both commit conflicting entries. A 5-node cluster tolerates 2 simultaneous failures; 3 failures drop the surviving 2 below the majority threshold and halt progress until recovery. The next lesson covers how the leader actually replicates writes to followers. Now when you see etcd report “leader changed” or “no leader elected,” you know to check which of these three invariants — one leader per term, majority quorum, term monotonicity — was violated and why.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
appears again in204
- Why GraphQL gets N+1junior
- DataLoader mechanics: tick-boundary batchingmiddle
- Batch function contracts: ordering, shapes, errorsmiddle
- Federation and lookahead: batching beyond DataLoadermiddle
- Query complexity defences: depth, cost, persisted queriesmiddle
- Senior GraphQL API: scheduling contract, tenant isolation, observabilitysenior
- Why idempotency: making retries safejunior
- Server-side state machine: four states of an idempotency keymiddle
- Outbox and inbox: effectively-once across the dual-write boundarymiddle
- Concurrency and cache architecture for idempotency at scalesenior
- Observability, production failures, and global-scale designsenior
- The event loop: one thread, three queuesjunior
- Tasks, microtasks, and scheduler.yield()middle
- Microtask starvation, Long Tasks, and LoAFsenior
- Node.js event loop: phases, nextTick, and loop lagsenior
- React, Vue, and INP observability in productionsenior
- The render pipeline: six stages from bytes to pixelsjunior
- Stage costs and the renderer process modelmiddle
- Invalidation, dirty bits, and containmiddle
- Compositor layers: promotion, overlap, and GPU memorymiddle
- DevTools flame strip and the frame lifecyclemiddle
- Layout thrash: forced synchronous layoutsenior
- BeginMainFrame, compositor-driven animations, and GPU memorysenior
- Production observability: LoAF, INP, and the full attack surfacesenior
- What V8 is and why performance varies 100×junior
- V8''''s four-tier JIT pipeline and profile-guided tieringmiddle
- Hidden classes, transition trees, and memory layoutmiddle
- Inline caches, IC states, and deoptimizationmiddle
- Orinoco GC: parallel scavenger, concurrent marking, and write barriersmiddle
- TurboFan''''s speculative engine and the deopt-loop trapsenior
- V8 in production: isolates, pointer compression, and real failuressenior
- Service worker lifecycle and cache strategiesmiddle
- Service worker edge cases: version skew, durability, and navigation trapssenior
- What the reconciler does: render vs commitjunior
- The fiber object and the double-buffer treemiddle
- Render phase purity and commit phase sub-stepsmiddle
- Reconciliation: diffing heuristics and the key trapmiddle
- Priority lanes, time-slicing, and useTransitionmiddle
- Bailout, memoisation, and tearingsenior
- React Profiler, the Compiler, and production observabilitysenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Hydration mismatch: causes, detection, and the determinism rulesenior
- RSC, per-route strategy, and production observabilitysenior
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- CLS: why layout shifts happen and how to stop themmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What is a cache stampede and why it makes things worsejunior
- Lock and single-flight: bounding concurrent rebuildsmiddle
- XFetch: coordination-free probabilistic early expirationmiddle
- Stale-while-revalidate and CDN request coalescingmiddle
- Detecting stampedes and designing TTL for productionmiddle
- Metastable failure, fencing tokens, and production postmortemssenior
- What a relation is: tables, rows, keys, and constraintsjunior
- Constraints, keys, and Postgres data typesmiddle
- Normal forms, denormalization, and why schemas stickmiddle
- JSONB, arrays, and when a side table winsmiddle
- Heap storage, TOAST, and column alignmentsenior
- Schema integrity: deferral, versioning, and production failure modessenior
- Relational vs document, wide-column, graph, and key-valuesenior
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- pg_statistic, ANALYZE, and production observabilitymiddle
- Production failure modes and plan stabilitysenior
- MVCC: why readers and writers never wait for each otherjunior
- Row versions and snapshots: the on-disk mechanicsmiddle
- HOT updates and isolation levels: what you gain and what you paymiddle
- Vacuum and bloat: keeping the storage tax boundedmiddle
- CLOG, XID wraparound, and MultiXact: deep visibility internalssenior
- SSI internals and production autovacuum tuningsenior
- Real-world MVCC failures, deployment patterns, and distributed snapshotssenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- What a schema migration is and why it replaces ad-hoc DDLjunior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Expand-contract: zero-downtime for breaking schema changesmiddle
- Advisory locks, migration tools, and deploy coordinationsenior
- Migration failure taxonomy and production disciplinesenior
- Why sharding exists: the single-Postgres ceilingjunior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Partitioning vs sharding: same word, two different thingsmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Schema-based sharding and multi-tenancy alternativessenior
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Where data fetching happens — and why it decides LCPjunior
- Fetch waterfalls — diagnosis and the Promise.all curemiddle
- React Server Components and Suspense streamingmiddle
- Client-side cache: TanStack Query, SWR, and stale-while-revalidatemiddle
- LCP, prefetch, and race conditions in interactive fetchingmiddle
- Senior internals: RSC payload, caching layers, and production failure modessenior
- What a JavaScript engine iszero
- Ignition and the bytecodemiddle
- Inside the interpreter loopmiddle
- How V8 represents a valuemiddle
- What a Map really is: opening the hidden classmiddle
- Transition trees, deprecation, and migrationmiddle
- Fast properties, slow properties, and slack trackingmiddle
- Inline caches, deeply: feedback slots and handlersmiddle
- Monomorphic, polymorphic, megamorphic — and the stub cachemiddle
- Four tiers and how warm-up worksmiddle
- Type feedback: the fuel for the optimisersenior
- TurboFan: the sea-of-nodes optimisersenior
- Speculation and guardssenior
- Deoptimization: falling off the cliffsenior
- On-Stack Replacement: swapping the frame mid-loopsenior
- Scopes, contexts, and the scope chainmiddle
- What a closure actually retainsmiddle
- When V8 allocates a Context (and what it costs)middle
- Closures, feedback vectors, and call-site polymorphismmiddle
- Why a GC, and reachabilitymiddle
- Leaks in a garbage-collected languagesenior
- The engine vs the loopmiddle
- Microtasks vs macrotasks: orderingmiddle
- Inside a Promisemiddle
- async/await, desugaredmiddle
- Capstone: optimize a hot pathsenior
- The three-way handshakejunior
- Sequence numbers and connection statemiddle
- DNS: what it does and why it existsjunior
- The resolver walk: referrals, record types, and gluemiddle
- TTL, caching, and DNS propagationmiddle
- The 1-RTT handshake: key shares and ECDHEmiddle
- Session resumption and 0-RTTmiddle
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket frame format: opcodes, masking, fragmentationmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Health checks, connection draining, and slow startmiddle
- Session affinity, consistent hashing, and the right fixmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- Connection IDs and network migrationmiddle
- 0-RTT resumption and packet encryptionsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- What the three signals are: logs, metrics, and tracesjunior
- Why structured logs exist: the diary vs the spreadsheetjunior
- The production log schema: fields every line must carrymiddle
- PII redaction and log injectionsenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- SLI, SLO, and the error budget: reliability by the numbersjunior
- Error budget policy, latency SLOs, and composite journeysmiddle
- Production SLO failures, self-observability, security, and the big picturesenior
- The incident loop: from pager to postmortem to preventionmiddle
- Cache lines, struct layout, and false sharingmiddle
- SIMD, SoA vs AoS, and memory bandwidthmiddle
- Cache-oblivious algorithms, PGO, and production failuressenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior
- At-most-once, at-least-once, exactly-once: the three delivery contractsjunior
- The three failure legs — where duplicates and losses actually happenmiddle
- Consumer-side dedup: the cheapest path to exactly-once processingmiddle
- Kafka exactly-once semantics: idempotent producer and transactionsmiddle
- SQS visibility timeout, DLQ, and the outbox patternmiddle
- Exactly-once in production: impossibility proof, hybrid patterns, and real incidentssenior
- What OAuth is and why passwords are not the answerjunior
- Authorization code flow with PKCEmiddle
- ID token validation and JWKS cache managementmiddle
- Refresh token rotation and scope-based least privilegemiddle
- Sender-constrained tokens: DPoP and mTLSsenior
- OAuth in production: audience attacks, observability, and real failuressenior
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.