Quorums: the R + W > N invariant and how it quietly breaks

A payments team runs Cassandra at RF=3 and, chasing write latency, sets writes to consistency ONE (W=1) and reads to ONE (R=1). For months it is fast and fine. Then one node hiccups during a deploy. A user updates their payout account; the single ack came from a node that died seconds later before replicating. The next read hits a different replica and serves the old account number. The write was acknowledged, durable to nobody, and silently gone. R=1 + W=1 means R + W = 2, and 2 is not greater than 3 — there was never a guarantee the read and the write touched the same node.

The overlap invariant is the whole game

Replicate one key to N nodes. A write succeeds when W of them acknowledge it; a read succeeds when R of them respond and you take the newest version. The single fact that makes this useful: if R + W > N, the set of nodes that acked the write and the set that answered the read must share at least one node by the pigeonhole principle — and that shared node carries the latest value. That is the entire guarantee. There is no magic, no consensus log, no leader: just two subsets of N forced to intersect.

Flip it around and the failure is just as mechanical. If R + W <= N, the two sets can be disjoint. A read is then allowed to pick R nodes that all missed the last write, and it will hand back stale data while believing it succeeded. Nothing logged an error, because nothing was wrong by the rules you configured — you just configured away the guarantee.

The canonical strong-consistency setting at N=3 is R=2, W=2: 2 + 2 = 4 > 3, so every read overlaps every write by one node. This is exactly what Cassandra’s QUORUM / LOCAL_QUORUM (floor(N/2) + 1 = 2 at RF=3) gives you, and what DynamoDB calls a strongly consistent read.

Tunable consistency: you pick the corner of the triangle

R and W are knobs, often per-query, not a global mode. That is the point of “tunable consistency” — the same N=3 cluster can serve a W=1 fire-and-forget metrics write and a W=2,R=2 strongly-consistent balance read in the same process. You are choosing, per operation, where you land on the consistency/availability/latency triangle.

Two extremes show why nobody runs them in production. W=N (write to ALL) gives the strongest write but kills write availability: with W=3 at RF=3, a single node being down — a routine deploy, a GC pause, a slow disk — fails every write, because all three must ack. Symmetrically R=N blocks all reads on one slow node. Quorum (W=2,R=2) is the sweet spot precisely because it tolerates one node down on both paths while still guaranteeing overlap. That single-node-failure tolerance is why “quorum” became the default mental model rather than ALL.

Sloppy quorum + hinted handoff: trading the invariant for uptime

Strict quorum has a hard edge: if fewer than W of the designated replicas (the preference list) are reachable, the write fails. During a network partition or a multi-node outage that means downtime. Dynamo’s answer, inherited by Cassandra and DynamoDB, is the sloppy quorum: when a home replica is unreachable, the write goes to the next healthy node in the ring instead, which stores it as a hint — a parcel tagged “this really belongs to node X.” When X comes back, the holder replays the hint to it (hinted handoff) and deletes its copy.

This keeps writes flowing through failures, and that availability is the entire reason it exists. But read the fine print: the write was accepted by nodes that are not in the normal read set. A reader doing a strict R against the home replicas can completely miss a hinted write — so during the failure window, sloppy quorum does not satisfy R + W > N against the canonical N. The overlap guarantee is suspended exactly when you most want it. Worse, if the node holding the hint dies before handoff completes and you were at low durability, that write is simply lost, recoverable only by a slower anti-entropy repair if at all.

The production failure modes, and the repair that hides them

Three ways this bites in prod, all subtle because nothing errors:

1. W=1 lost write. The lone acking node fails before replicating. The write was “successful” and is now on zero live replicas. This is the Hook.
2. Sloppy-quorum stale read. During a partition, the write landed on hint-holders; a strict read on the home replicas serves the old value. Reads look healthy on both sides of the partition.
3. R + W <= N drift. R=1,W=2 (or any sub-overlap mix) lets a read pick the one replica that lagged. Intermittent, unreproducible “ghost stale data” tickets.

Two anti-entropy mechanisms paper over the gaps so the system is eventually consistent. Read repair: when a quorum read notices replicas disagree, the coordinator pushes the newest version to the stale ones inline — cheap, but only fixes keys you actually read. Anti-entropy repair (nodetool repair): replicas build Merkle trees, hash-tree diffs, and stream only mismatched ranges to reconcile data nobody has read. Skip scheduled repair within the tombstone GC window (gc_grace_seconds, default 10 days) and deleted data can resurrect — a separate, nasty class of bug.

Latency: a quorum read is only as fast as its slowest required replica

R is not just a correctness knob, it is a tail-latency knob. A QUORUM read at RF=3 waits for 2 of 3 replicas, so its latency is the second-fastest of three — meaning one slow replica (GC pause, hot partition, throttled disk) drags the read’s p99 even though the cluster as a whole is fine. Raising R to ALL makes you wait on the slowest of all replicas, multiplying tail risk. This is why DynamoDB strongly-consistent reads cost ~2x the throughput of eventually-consistent ones and run measurably slower, and why systems use request hedging — fire a duplicate request after a short delay and take whichever returns first — to clip the p99 that quorum’s “wait for the slowest needed replica” otherwise creates.