Data distribution: shard, replicate, and break it
Hands-on project: take a dataset, design its partition key and replication, build a tiny sharded + consistent-hashed store, then deliberately trigger a hot shard, a lost write on failover, and a quorum-staleness read — and prove the mitigations work.
Reading that async failover loses writes and that mod-N wrecks a reshard is not the same as watching it happen on a store you built. Take a dataset, design its partition key and replication, implement a tiny sharded and consistent-hashed key-value store, and then deliberately reproduce the unit’s three signature failures — the hot shard, the lost write, and the stale quorum read — and prove your mitigations actually close each one.
This project makes the whole unit operational: you’ll commit to a partition key and a replication mode, route keys with a hash ring, and then trigger each failure on purpose so the mitigation is something you measured, not something you read.
Design and build a tiny distributed key-value store (any language, in-process or multi-process) that shards data by a chosen partition key, routes keys with consistent hashing, and replicates each shard — then deliberately reproduce a hot shard, a lost write on failover, and a stale quorum read, and demonstrate the mitigation for each.
- A one-paragraph brief with the dominant query, read:write ratio, and key cardinality, plus a justified partition key chosen from the access pattern.
- A measurement showing consistent hashing moves ~1/N of keys on a membership change versus a mod-N baseline that moves nearly all — with the actual fractions reported.
- A reproduced hot shard (one shard's load far above the rest under skew) and an after measurement showing the mitigation spread the load.
- A reproduced lost write under async failover, and a re-run under semi-sync where the same acknowledged write survives — with the exact sequence that produced each outcome.
- A quorum demonstration: a stale read at R + W ≤ N and a guaranteed-fresh read at R + W > N, with the N, W, R values used.
- A short architecture-decisions note tying each choice (key, replication mode, quorum, CAP/PACELC point) to the specific guarantee or trade it buys.
- Add split-brain: partition the cluster, let both sides accept writes, then heal and show the divergent histories — and implement fencing or quorum promotion that prevents the second leader from accepting writes.
- Add a resharding path: grow the ring from N to N+2 nodes under live writes using double-writes or change-data-capture, and show no acknowledged write is lost during the cutover.
- Add bounded-load consistent hashing (cap each node at c×average and spill clockwise) and show it tames the hot shard for request routing while preserving the ~1/N movement property.
- Add a multi-leader mode with two regions accepting writes, inject a conflicting concurrent write, and implement a conflict resolution (last-write-wins vs a version vector) — discussing which silently loses data.
- 01How do you demonstrate that consistent hashing beats mod-N on a membership change?
- 02What's the exact sequence that produces a lost write under async failover, and how does semi-sync fix it?
- 03How do you show a quorum guarantees (or fails to guarantee) a fresh read?
This project turns the data-distribution unit into something you’ve operated, not just read. You choose a partition key from the dominant access pattern and cardinality, route keys on a consistent-hashing ring with virtual nodes (and measure the ~1/N movement against a mod-N baseline that remaps nearly all), and replicate each shard in both synchronous and asynchronous modes. Then you reproduce the unit’s three signature failures on purpose and close each: a hot shard under skew (mitigated by splitting, caching, or isolating the celebrity key), a lost write under async failover (fixed by semi-sync so an ack means ‘on two nodes’), and a stale quorum read at R + W ≤ N (fixed by R + W > N so the read and write sets must overlap). The architecture-decisions note ties every choice — key, replication mode, quorum, CAP/PACELC point — to the specific guarantee it buys. An engineer who has built and broken this store once distributes data deliberately, choosing which disagreement to tolerate up front, instead of discovering a lost write or a wiped cache during a 3 a.m. failover.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.