Distributed Systems
Quorums: config and code reading
Crux Read real consistency-level configs and a quorum overlap check, predict whether each guarantees the latest write, and pick the highest-leverage fix.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitQuorum bugs live in the consistency-level lines of your data-access code, not in the database. Read each config and snippet, do the R + W vs N arithmetic, and choose the fix a senior engineer would make first.
Goal
Practise the loop you run when a stale-read ticket lands: find the R and W in the code, check whether R + W > N actually holds for the keyspace’s replication factor, and reach for the fix that restores overlap rather than one that hides it.
Snippet 1 — the read/write consistency split
# keyspace created WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': 3 }
session.execute(
SimpleStatement(
"UPDATE accounts SET payout = %s WHERE id = %s",
consistency_level=ConsistencyLevel.QUORUM, # W = 2 at RF=3
),
(new_payout, account_id),
)
balance = session.execute(
SimpleStatement(
"SELECT payout FROM accounts WHERE id = %s",
consistency_level=ConsistencyLevel.ONE, # R = 1
),
(account_id,),
).one() Quiz
Completed
The write is QUORUM (W=2) and the read is ONE (R=1) at RF=3. Does this guarantee the read sees the latest payout, and what is the fix?
Heads-up A QUORUM write is on 2 of 3, but R=1 can read the remaining 1 that lagged. You need R + W > N (forced intersection), not just a majority write. 1 + 2 = 3 does not satisfy > 3.
Heads-up W=3 with R=1 sums to 4 > 3 and would work, but write-ALL fails every write on a single node down — terrible availability for a payments write. Raising the READ to QUORUM keeps single-node tolerance on both paths.
Heads-up The coordinator routes the read to replicas; it is not guaranteed to be a replica for this key or to hold the latest version. Overlap must be guaranteed by R + W > N.
Snippet 2 — the overlap check helper
def guarantees_latest_read(n: int, r: int, w: int) -> bool:
# returns True iff a read is guaranteed to see the latest committed write
return r + w > n
# called for a new keyspace plan, RF = 4
print(guarantees_latest_read(n=4, r=2, w=2)) # what prints, and is it safe? Quiz
Completed
At N=4 with R=2, W=2, what does guarantees_latest_read print, and is that the right answer?
Heads-up 2 is not a majority of 4 — a strict majority of 4 is 3. And the helper uses strict >, so 2 + 2 = 4 returns False. Even-N clusters are a classic trap: a 'half' read and 'half' write need not intersect.
Heads-up Strict > is exactly right. At R + W = N the two sets can be perfectly disjoint (e.g. nodes {1,2} write, {3,4} read at N=4), so >= would wrongly report safety. The pigeonhole intersection needs R + W > N.
Heads-up A higher RF does not by itself give overlap; the R/W choice does. R=2,W=2 at N=4 sums to 4, fails the > N test, and permits stale reads regardless of how many replicas exist.
Snippet 3 — the sloppy-quorum write path
def write_quorum(coordinator, key, value, n=3, w=2):
home = coordinator.preference_list(key, n) # the N designated replicas
live = [node for node in home if node.is_reachable()]
acks = [node.store(key, value) for node in live]
if len(acks) >= w:
return "OK (strict quorum)"
# not enough designated replicas reachable -> sloppy quorum
for sub in coordinator.next_healthy_ring_nodes():
acks.append(sub.store_hint(key, value, intended_owner=...))
if len(acks) >= w:
return "OK (sloppy quorum)" # acked, but where?
return "FAIL" Quiz
Completed
When this returns 'OK (sloppy quorum)' during a partition, what is the consistency consequence for a concurrent strict QUORUM read on the home replicas?
Heads-up store_hint writes to SUBSTITUTE ring nodes (next_healthy_ring_nodes), not the designated replicas. A read of the home preference list can miss the hint entirely until hinted handoff completes.
Heads-up Hints do not block or fail reads of the home replicas; those replicas simply serve their (older) version. The read succeeds and returns stale data — the silent failure mode.
Heads-up There is no ordering guarantee between handoff and a racing read. During the partition window the read can land before handoff, which is exactly the stale-read seam sloppy quorum opens.
Snippet 4 — the DynamoDB read flag
# DynamoDB, replication is managed; you pick read consistency per request
resp = table.get_item(
Key={"id": account_id},
ConsistentRead=False, # default: eventually consistent
) Quiz
Completed
A balance read uses ConsistentRead=False. What does flipping it to True buy you, and what does it cost — in quorum terms?
Heads-up It changes the data you can observe: False may return a value from a replica that hasn't yet received a recent write (stale), while True is guaranteed to reflect all prior acked writes. It also costs ~2x RCU.
Heads-up ConsistentRead is a READ flag and does not change write durability at all. It only governs whether a read is guaranteed to see the latest committed write.
Heads-up It costs about double the read capacity and adds latency, so you reserve it for reads that must not be stale (balances, idempotency checks) and leave loss-tolerant reads eventually-consistent — the tunable-consistency tradeoff.
Recap
Every quorum bug is read the same way: find R and W in the consistency-level lines, find N from the keyspace replication factor, and check R + W > N with a STRICT inequality — R=1 reads behind QUORUM writes, and any even-N split like R=2,W=2 at N=4, are the classic traps that sum to N and silently permit stale reads. The fix that restores overlap (usually raising the READ level) beats fixes that just hide it. Sloppy quorum acks via substitute hint-holders outside the read set, so its ‘OK’ suspends overlap during a partition; and managed stores expose the same dial as ConsistentRead, where strong reads cost roughly 2x capacity for the guarantee.