Distributed Systems
Raft extensions: pre-vote, learners, snapshots, and linearizable reads
Your etcd cluster has 50,000 reads per second but only 200 writes per second. If every read triggers a consensus round, the cluster spends 250x more work on reads than writes. Yet if you serve reads from a stale follower, you lose linearizability. How do production Raft implementations serve reads at sub-millisecond latency without sacrificing correctness?
Pre-vote: preventing spurious elections
When a node returns from a long partition, it has been incrementing its term counter the whole time. Its RequestVote arrives with a very high term, forcing healthy followers to update their terms and step down the current leader — even though no actual failure occurred. The cluster suffers an unnecessary election.
Pre-vote adds a non-binding dry run before any real election. Before incrementing its term, a candidate sends a PreVote message. Nodes respond “I would vote for you” or “I would not” without changing any persistent state. Only if a majority would vote yes does the candidate actually increment and run a real election.
Cost: one extra RPC round when an election is genuinely needed. Benefit: zero spurious leader changes from rejoining nodes. Pre-vote is now standard in etcd, TiKV, and Hashicorp Raft.
Learners: safe cluster scale-out
Adding a node directly as a voter is dangerous: the new node starts with an empty log and its slow catch-up degrades commit latency for every write until it is current. In a 3-node cluster adding a 4th voter also temporarily drops fault-tolerance (quorum is now 3 of 4 instead of 2 of 3).
Learners (non-voting members) solve this. A learner receives AppendEntries and replicates the log like a follower, but does not count toward quorums and does not vote in elections. The operator promotes the learner to voter only once its log lag is below a threshold (typically under 1000 entries behind). The promotion is then a single-server membership change — safe, fast, and transparent.
Learners are also how etcd, Consul, and Yugabyte handle node replacement after a failure: the new node learns, catches up, gets promoted, then the dead node is removed. Availability never drops below the original quorum during the replacement.
Snapshots and log compaction
A Raft log grows indefinitely. A one-year-old cluster with 10k writes/s has written 315 billion entries — the log would be terabytes, and replaying it after a crash would take days.
Snapshots compact the log: periodically, each node serialises its full state machine state plus the (lastIncludedIndex, lastIncludedTerm) tuple to disk, then deletes all log entries up to that index. If a follower falls so far behind that the leader has already compacted the entries it needs, the leader sends an InstallSnapshot RPC instead of the missing AppendEntries — the follower loads the snapshot as its starting state, then replays only the recent log tail.
| System | Default snapshot interval |
|---|---|
| etcd | Every 10,000 entries |
| CockroachDB | Every few seconds per range |
| TiKV | Configurable, default 200 MB |
Snapshot frequency is a tunable tradeoff: too often amplifies fsync; too rare grows the log and slows recovery. The snapshot must include the latest membership configuration — forgetting this is a well-known bug in DIY Raft implementations.
ReadIndex: linearizable reads without writing to the log
A naive approach to linearizable reads: the leader commits a no-op log entry just for the read, then returns data. This serialises reads into the log — expensive (one full consensus round per read).
ReadIndex eliminates the log write. When a read arrives:
- Leader records its current
commitIndex. - Leader sends a heartbeat to a majority of followers to confirm it is still the active leader (prevents a stale leader from serving reads after a partition).
- Once its state machine has applied entries up to
commitIndex, the leader returns the result.
Cost: one heartbeat round-trip (typically 1–5 ms intra-region). No new log entry. Reads scale independently of write throughput.
Leader lease reads: sub-millisecond latency
ReadIndex still pays a network round-trip per read. Lease reads go further: the leader holds a lease — a time window during which it is guaranteed to be the only leader (because no election can complete within the lease duration). During the lease, the leader serves reads from its current state machine without any RPC.
Lease duration is typically election_timeout * 0.9 (e.g., 9 ms for a 10 ms election timeout), leaving a 10% buffer for clock skew.
The correctness condition: lease reads are safe only if the clock skew between leader and followers is bounded. If the leader’s clock runs significantly faster than any follower’s, the lease may expire on a follower before the leader thinks it has — a follower could start a new election and elect a new leader while the old leader is still serving reads. This is why NTP/PTP synchronisation is a correctness requirement for lease reads, not just a hygiene practice.
- fsync latency, NVMe + BBU
- 50–100 µs
- fsync latency, cloud SSD (EBS gp3)
- 1–3 ms
- ReadIndex latency (intra-region)
- 1–5 ms
- Lease read latency
- under 1 ms
- Etcd snapshot interval (default)
- 10,000 entries
- TimeoutNow leadership transfer
- under 10 ms unavailability
Why is leader lease read considered correctness-sensitive while ReadIndex is not?
A new node is added to a 3-node Raft cluster directly as a voter. Why might this temporarily degrade commit latency?
- 01A node rejoins after 30 minutes offline. Without pre-vote, what happens to the cluster?
- 02What information must a Raft snapshot include, beyond the serialised state machine?
- 03ReadIndex is described as requiring 'one heartbeat round-trip per read.' Why cannot the leader skip this and serve reads immediately from its current state?
Four extensions complete the gap between textbook Raft and production systems. Pre-vote eliminates spurious elections by requiring a dry-run before term increment. Learners allow safe scale-out by letting new nodes catch up before they count toward quorum. Snapshots bound log growth by periodically checkpointing state machine state and allowing InstallSnapshot for lagging followers. ReadIndex enables linearizable reads with one heartbeat round-trip instead of a full log write; lease reads take this to sub-millisecond by time-bounding leader authority, but require NTP synchronisation as a correctness precondition. All four are standard in etcd, TiKV, and Hashicorp Raft.
appears again in178
- Why GraphQL gets N+1junior
- DataLoader mechanics: tick-boundary batchingmiddle
- Batch function contracts: ordering, shapes, errorsmiddle
- Federation and lookahead: batching beyond DataLoadermiddle
- Query complexity defences: depth, cost, persisted queriesmiddle
- Senior GraphQL API: scheduling contract, tenant isolation, observabilitysenior
- Why idempotency: making retries safejunior
- Server-side state machine: four states of an idempotency keymiddle
- Outbox and inbox: effectively-once across the dual-write boundarymiddle
- Concurrency and cache architecture for idempotency at scalesenior
- Observability, production failures, and global-scale designsenior
- The event loop: one thread, three queuesjunior
- Tasks, microtasks, and scheduler.yield()middle
- Microtask starvation, Long Tasks, and LoAFsenior
- Node.js event loop: phases, nextTick, and loop lagsenior
- React, Vue, and INP observability in productionsenior
- The render pipeline: six stages from bytes to pixelsjunior
- Stage costs and the renderer process modelmiddle
- Invalidation, dirty bits, and containmiddle
- Compositor layers: promotion, overlap, and GPU memorymiddle
- DevTools flame strip and the frame lifecyclemiddle
- Layout thrash: forced synchronous layoutsenior
- BeginMainFrame, compositor-driven animations, and GPU memorysenior
- Production observability: LoAF, INP, and the full attack surfacesenior
- What V8 is and why performance varies 100×junior
- V8''''s four-tier JIT pipeline and profile-guided tieringmiddle
- Hidden classes, transition trees, and memory layoutmiddle
- Inline caches, IC states, and deoptimizationmiddle
- Orinoco GC: parallel scavenger, concurrent marking, and write barriersmiddle
- TurboFan''''s speculative engine and the deopt-loop trapsenior
- V8 in production: isolates, pointer compression, and real failuressenior
- Service worker lifecycle and cache strategiesmiddle
- Service worker edge cases: version skew, durability, and navigation trapssenior
- What the reconciler does: render vs commitjunior
- The fiber object and the double-buffer treemiddle
- Render phase purity and commit phase sub-stepsmiddle
- Reconciliation: diffing heuristics and the key trapmiddle
- Priority lanes, time-slicing, and useTransitionmiddle
- Bailout, memoisation, and tearingsenior
- React Profiler, the Compiler, and production observabilitysenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Hydration mismatch: causes, detection, and the determinism rulesenior
- RSC, per-route strategy, and production observabilitysenior
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- CLS: why layout shifts happen and how to stop themmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What is a cache stampede and why it makes things worsejunior
- Lock and single-flight: bounding concurrent rebuildsmiddle
- XFetch: coordination-free probabilistic early expirationmiddle
- Stale-while-revalidate and CDN request coalescingmiddle
- Detecting stampedes and designing TTL for productionmiddle
- Metastable failure, fencing tokens, and production postmortemssenior
- What a relation is: tables, rows, keys, and constraintsjunior
- Constraints, keys, and Postgres data typesmiddle
- Normal forms, denormalization, and why schemas stickmiddle
- JSONB, arrays, and when a side table winsmiddle
- Heap storage, TOAST, and column alignmentsenior
- Schema integrity: deferral, versioning, and production failure modessenior
- Relational vs document, wide-column, graph, and key-valuesenior
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- pg_statistic, ANALYZE, and production observabilitymiddle
- Production failure modes and plan stabilitysenior
- MVCC: why readers and writers never wait for each otherjunior
- Row versions and snapshots: the on-disk mechanicsmiddle
- HOT updates and isolation levels: what you gain and what you paymiddle
- Vacuum and bloat: keeping the storage tax boundedmiddle
- CLOG, XID wraparound, and MultiXact: deep visibility internalssenior
- SSI internals and production autovacuum tuningsenior
- Real-world MVCC failures, deployment patterns, and distributed snapshotssenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- What a schema migration is and why it replaces ad-hoc DDLjunior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Expand-contract: zero-downtime for breaking schema changesmiddle
- Advisory locks, migration tools, and deploy coordinationsenior
- Migration failure taxonomy and production disciplinesenior
- Why sharding exists: the single-Postgres ceilingjunior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Partitioning vs sharding: same word, two different thingsmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Schema-based sharding and multi-tenancy alternativessenior
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Where data fetching happens — and why it decides LCPjunior
- Fetch waterfalls — diagnosis and the Promise.all curemiddle
- React Server Components and Suspense streamingmiddle
- Client-side cache: TanStack Query, SWR, and stale-while-revalidatemiddle
- LCP, prefetch, and race conditions in interactive fetchingmiddle
- Senior internals: RSC payload, caching layers, and production failure modessenior
- The three-way handshakejunior
- Sequence numbers and connection statemiddle
- DNS: what it does and why it existsjunior
- The resolver walk: referrals, record types, and gluemiddle
- TTL, caching, and DNS propagationmiddle
- The 1-RTT handshake: key shares and ECDHEmiddle
- Session resumption and 0-RTTmiddle
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket frame format: opcodes, masking, fragmentationmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Health checks, connection draining, and slow startmiddle
- Session affinity, consistent hashing, and the right fixmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- Connection IDs and network migrationmiddle
- 0-RTT resumption and packet encryptionsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- What the three signals are: logs, metrics, and tracesjunior
- Why structured logs exist: the diary vs the spreadsheetjunior
- The production log schema: fields every line must carrymiddle
- PII redaction and log injectionsenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- SLI, SLO, and the error budget: reliability by the numbersjunior
- Error budget policy, latency SLOs, and composite journeysmiddle
- Production SLO failures, self-observability, security, and the big picturesenior
- The incident loop: from pager to postmortem to preventionmiddle
- Cache lines, struct layout, and false sharingmiddle
- SIMD, SoA vs AoS, and memory bandwidthmiddle
- Cache-oblivious algorithms, PGO, and production failuressenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior
- At-most-once, at-least-once, exactly-once: the three delivery contractsjunior
- The three failure legs — where duplicates and losses actually happenmiddle
- Consumer-side dedup: the cheapest path to exactly-once processingmiddle
- Kafka exactly-once semantics: idempotent producer and transactionsmiddle
- SQS visibility timeout, DLQ, and the outbox patternmiddle
- Exactly-once in production: impossibility proof, hybrid patterns, and real incidentssenior
- What OAuth is and why passwords are not the answerjunior
- Authorization code flow with PKCEmiddle
- ID token validation and JWKS cache managementmiddle
- Refresh token rotation and scope-based least privilegemiddle
- Sender-constrained tokens: DPoP and mTLSsenior
- OAuth in production: audience attacks, observability, and real failuressenior