awesome-everything RU
↑ Back to the climb

Distributed Systems

Raft extensions: pre-vote, learners, snapshots, and linearizable reads

Crux Four production-critical Raft extensions — pre-vote to prevent term inflation, learners for safe scale-out, snapshots for log compaction, and ReadIndex/lease reads for fast linearizable queries.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbit
◷ 16 min

Your etcd cluster has 50,000 reads per second but only 200 writes per second. If every read triggers a consensus round, the cluster spends 250x more work on reads than writes. Yet if you serve reads from a stale follower, you lose linearizability. How do production Raft implementations serve reads at sub-millisecond latency without sacrificing correctness?

Pre-vote: preventing spurious elections

When a node returns from a long partition, it has been incrementing its term counter the whole time. Its RequestVote arrives with a very high term, forcing healthy followers to update their terms and step down the current leader — even though no actual failure occurred. The cluster suffers an unnecessary election.

Pre-vote adds a non-binding dry run before any real election. Before incrementing its term, a candidate sends a PreVote message. Nodes respond “I would vote for you” or “I would not” without changing any persistent state. Only if a majority would vote yes does the candidate actually increment and run a real election.

Cost: one extra RPC round when an election is genuinely needed. Benefit: zero spurious leader changes from rejoining nodes. Pre-vote is now standard in etcd, TiKV, and Hashicorp Raft.

Learners: safe cluster scale-out

Adding a node directly as a voter is dangerous: the new node starts with an empty log and its slow catch-up degrades commit latency for every write until it is current. In a 3-node cluster adding a 4th voter also temporarily drops fault-tolerance (quorum is now 3 of 4 instead of 2 of 3).

Learners (non-voting members) solve this. A learner receives AppendEntries and replicates the log like a follower, but does not count toward quorums and does not vote in elections. The operator promotes the learner to voter only once its log lag is below a threshold (typically under 1000 entries behind). The promotion is then a single-server membership change — safe, fast, and transparent.

Learners are also how etcd, Consul, and Yugabyte handle node replacement after a failure: the new node learns, catches up, gets promoted, then the dead node is removed. Availability never drops below the original quorum during the replacement.

Snapshots and log compaction

A Raft log grows indefinitely. A one-year-old cluster with 10k writes/s has written 315 billion entries — the log would be terabytes, and replaying it after a crash would take days.

Snapshots compact the log: periodically, each node serialises its full state machine state plus the (lastIncludedIndex, lastIncludedTerm) tuple to disk, then deletes all log entries up to that index. If a follower falls so far behind that the leader has already compacted the entries it needs, the leader sends an InstallSnapshot RPC instead of the missing AppendEntries — the follower loads the snapshot as its starting state, then replays only the recent log tail.

SystemDefault snapshot interval
etcdEvery 10,000 entries
CockroachDBEvery few seconds per range
TiKVConfigurable, default 200 MB

Snapshot frequency is a tunable tradeoff: too often amplifies fsync; too rare grows the log and slows recovery. The snapshot must include the latest membership configuration — forgetting this is a well-known bug in DIY Raft implementations.

ReadIndex: linearizable reads without writing to the log

A naive approach to linearizable reads: the leader commits a no-op log entry just for the read, then returns data. This serialises reads into the log — expensive (one full consensus round per read).

ReadIndex eliminates the log write. When a read arrives:

  1. Leader records its current commitIndex.
  2. Leader sends a heartbeat to a majority of followers to confirm it is still the active leader (prevents a stale leader from serving reads after a partition).
  3. Once its state machine has applied entries up to commitIndex, the leader returns the result.

Cost: one heartbeat round-trip (typically 1–5 ms intra-region). No new log entry. Reads scale independently of write throughput.

Leader lease reads: sub-millisecond latency

ReadIndex still pays a network round-trip per read. Lease reads go further: the leader holds a lease — a time window during which it is guaranteed to be the only leader (because no election can complete within the lease duration). During the lease, the leader serves reads from its current state machine without any RPC.

Lease duration is typically election_timeout * 0.9 (e.g., 9 ms for a 10 ms election timeout), leaving a 10% buffer for clock skew.

The correctness condition: lease reads are safe only if the clock skew between leader and followers is bounded. If the leader’s clock runs significantly faster than any follower’s, the lease may expire on a follower before the leader thinks it has — a follower could start a new election and elect a new leader while the old leader is still serving reads. This is why NTP/PTP synchronisation is a correctness requirement for lease reads, not just a hygiene practice.

Raft extension numbers
fsync latency, NVMe + BBU
50–100 µs
fsync latency, cloud SSD (EBS gp3)
1–3 ms
ReadIndex latency (intra-region)
1–5 ms
Lease read latency
under 1 ms
Etcd snapshot interval (default)
10,000 entries
TimeoutNow leadership transfer
under 10 ms unavailability
Quiz

Why is leader lease read considered correctness-sensitive while ReadIndex is not?

Quiz

A new node is added to a 3-node Raft cluster directly as a voter. Why might this temporarily degrade commit latency?

Recall before you leave
  1. 01
    A node rejoins after 30 minutes offline. Without pre-vote, what happens to the cluster?
  2. 02
    What information must a Raft snapshot include, beyond the serialised state machine?
  3. 03
    ReadIndex is described as requiring 'one heartbeat round-trip per read.' Why cannot the leader skip this and serve reads immediately from its current state?
Recap

Four extensions complete the gap between textbook Raft and production systems. Pre-vote eliminates spurious elections by requiring a dry-run before term increment. Learners allow safe scale-out by letting new nodes catch up before they count toward quorum. Snapshots bound log growth by periodically checkpointing state machine state and allowing InstallSnapshot for lagging followers. ReadIndex enables linearizable reads with one heartbeat round-trip instead of a full log write; lease reads take this to sub-millisecond by time-bounding leader authority, but require NTP synchronisation as a correctness precondition. All four are standard in etcd, TiKV, and Hashicorp Raft.

Connected lessons
appears again in178
Continue the climb ↑Raft in production: membership changes, Multi-Raft, and observability
shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources4
expand
  1. 01
  2. 02
  3. 03
  4. 04

Trademarks belong to their respective owners. Editorial reference only.