Read real replication config, a partition-key choice, a routing function, and a quorum setting, then do the reasoning: spot the lost-write risk, the hot shard, the mod-N reshard trap, and whether a quorum guarantees a fresh read.
SDSenior◷ 14 min
Level
FoundationsJuniorMiddleSenior
Distribution bugs live in the config and the routing code: a replication mode, a partition key, a modulus, a quorum number. Read each snippet, reason through the failure it implies, and pick the answer a senior engineer would commit to in review.
Practise the loop you run during a design or incident review: locate the distribution decision in the config or code, trace its failure mode, and choose the change the mechanism actually supports.
Snippet 1 — the replication config
# database replicationreplication: mode: async # leader acks before followers receive the write followers: 3 sync_replicas: 0 # none required to ack# on leader failure: auto-promote the most-caught-up follower
Quiz
Completed
What durability risk does this config carry, and what's the minimal change?
Heads-up Promoting the most-caught-up follower minimizes loss but can't recover writes that reached only the dead leader. With sync_replicas: 0 those acked writes are gone. You need a synchronous replica so the ack itself implies durability.
Heads-up More followers scale reads but don't make an ack durable — they're all async here. The risk is LOST WRITES on failover, fixed by requiring a synchronous replica, not by adding async ones.
Heads-up All-synchronous couples write availability to the slowest of 3 followers — one slow replica stalls every write. The minimal safe change is sync_replicas: 1 (semi-sync): durable on two nodes without holding writes hostage to all three.
Snippet 2 — the partition key
# events table is sharded; choose the shard keydef shard_for(event): return hash(event.created_at_day) % NUM_SHARDS # shard by day# hottest query: SELECT ... WHERE tenant_id = ? AND created_at_day = ?
Quiz
Completed
What's wrong with sharding events by created_at_day, given the workload?
Heads-up Hashing the day spreads DIFFERENT days across shards, but all of one day's writes share a key, so today's writes all hit one shard — a write hot spot. And the query needs tenant_id, which the day-key can't route on.
Heads-up A prime modulus is irrelevant to this bug. The problem is the CHOICE of key (day) versus the access pattern (tenant + day): today's writes concentrate and the hot query can't be routed single-shard.
Heads-up A secondary index on a non-shard-key is local-per-shard (scatter-gather to query) or a separate global index — it doesn't fix the write hot spot from the day key. Re-keying to tenant_id addresses both the spread and the query.
Snippet 3 — the routing function
# cache client picks a node for each keydef node_for(key, nodes): return nodes[hash(key) % len(nodes)] # nodes is a live, autoscaling list# autoscaler adds/removes cache nodes based on load
Quiz
Completed
The cache fleet autoscales, so len(nodes) changes at runtime. What does this routing cause, and what's the fix?
Heads-up It doesn't: the divisor len(nodes) changes on every scale event, so hash(key) % len(nodes) reassigns nearly every key. That's the rehash-everything problem; mod-N is unsafe for a live, changing node set.
Heads-up The hash is fine; the problem is mod-N's dependence on len(nodes), which remaps everything when the count changes. A better hash won't help — consistent hashing (decoupling routing from N) will.
Heads-up Hard-coding the count breaks routing the moment the real node set differs from the constant (keys route to nodes that may not exist). The correct fix is consistent hashing, which handles a changing node set by design.
Snippet 4 — the quorum setting
# leaderless key-value storeN = 5 # replicas per keyW = 2 # nodes that must ack a writeR = 2 # nodes that must answer a read# product requirement: a read must always see the latest acknowledged write
Quiz
Completed
Does W = 2, R = 2 with N = 5 meet the requirement, and if not, what's the minimal correct setting?
Heads-up Only quorums with R + W > N guarantee overlap. Here R + W = 4 ≤ 5, so the two-node read set can entirely miss the two-node write set; the read can return stale data. You must raise the sum above N.
Heads-up W = 5 does guarantee fresh reads (any R overlaps), but it requires all replicas for a write — the least write availability, more than needed. R + W > 5 is enough; W = 3, R = 3 meets it with better write availability.
Heads-up Lowering R makes R + W = 3, still ≤ 5 and still non-overlapping — reads get faster AND less correct. To guarantee freshness you must increase the sum past N, e.g. W = 3, R = 3.
Recall before you leave
01
How do you read durability risk off a replication config?
02
Why is hash(key) % len(nodes) unsafe for an autoscaling node set, and what replaces it?
03
Given N, W, R, how do you tell if reads are guaranteed fresh, and what's the minimal fix when they aren't?
Recap
Every distribution decision in this unit is something you can read straight off the config or routing code. A replication config with mode: async and sync_replicas: 0 acks before any follower has the write, so a failover loses acknowledged writes — the fix is one synchronous replica. A partition key chosen against the workload (sharding events by day when the hot query filters by tenant) creates a write hot spot and an unroutable query — re-key to match the access pattern. A routing function of hash(key) % len(nodes) over an autoscaling node set remaps nearly all keys on every scale event — replace it with a consistent-hashing ring so only ~1/N move. And a quorum guarantees a fresh read only when R + W > N — W = 2, R = 2, N = 5 (sum 4) does not, while W = 3, R = 3 (sum 6) does. The senior habit is to find the distribution decision in the config, trace its failure mode, and choose the fix the mechanism supports — not the one that hides it.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.