Replication
Copy data to many nodes for reads, locality, and survival — but every copy is a lie until it catches up. Leader-follower, multi-leader, and leaderless each trade write availability against the chance of silently losing an acknowledged write.
A user updates their shipping address, sees the success toast, and refreshes — the old address is back. They update it again. Old address again. Support can’t reproduce it. What’s happening is invisible from any single console: the write landed on the leader, the success came back, and then the read was served by a follower that hadn’t received the change yet. No bug, no error, no exception. Just two copies of the truth that disagree for a few hundred milliseconds, and a customer who now thinks your product is broken. Replication is how you survive a node dying and serve reads at scale — and it is also where “I saved it” quietly stops being true. By the end of this lesson you’ll know why that happens, when acknowledged writes can vanish entirely, and which single routing change fixes the vanishing-address bug.
Why copy data at all
A single node holding your data is a single point of failure and a single throughput limit — the exact problem the previous unit’s vertical-scaling lesson ended on. Replication keeps the same data on multiple nodes, and it buys three distinct things, which people conflate at their peril:
- Read scaling — fan reads out across many copies instead of hammering one node.
- Locality — put a copy near the user (a replica in Europe for European reads) to cut latency.
- Fault tolerance — if one node dies, another holds the data and can take over.
Each of these is worth having — but they all come bundled with the same hidden cost: the moment there are two copies, they can disagree, and the system must decide which one to trust.
The catch is the one the hook just showed: the moment there is more than one copy, those copies can disagree. Replication is therefore not a feature you turn on; it is a consistency model you choose, and the choice is which disagreement you can tolerate.
Leader-follower (the default)
The most common topology is leader-follower (also called primary-replica or master-slave). One node is the leader: all writes go to it. The leader streams its changes — usually a replication log of every write — to one or more followers, which apply them in order. Reads can be served by the leader or any follower.
The replication can be synchronous or asynchronous, and this single switch is the heart of the lesson:
- Synchronous: the leader waits for a follower to confirm the write before acknowledging it to the client. The follower is guaranteed current — but if that follower is slow or down, the write blocks. Make all followers synchronous and one slow replica stalls every write.
- Asynchronous: the leader acknowledges the write immediately and ships it to followers in the background. Fast and available — but the followers trail behind, and a write the leader has acknowledged may not yet exist anywhere else.
Most production systems run semi-synchronous: one synchronous follower (so an acknowledged write survives on at least two nodes) and the rest asynchronous (so reads scale without blocking writes).
Replication lag and read-your-writes
The gap between “the leader has the write” and “this follower has the write” is replication lag. Under load, or across a region, it ranges from sub-millisecond to seconds — and during a long-running operation or a network hiccup it can blow out to minutes. Lag is not a bug; it is the price of asynchronous replication.
Lag produces a family of anomalies, and the hook’s address bug is the most common: read-your-writes (also called read-after-write consistency). A user makes a write, then immediately reads — and gets routed to a lagging follower that hasn’t applied it, so their own change appears to have vanished. The fixes are all about routing, not about making replication faster:
- Read a user’s own recently-written data from the leader for a short window after their write.
- Track the write’s log position (a version token) and route the read to a follower that has caught up to it.
- Pin a user to one replica (with the trade-offs sticky sessions always carry).
▸Why this works
Why not just make all replication synchronous and be done with it? Because synchronous replication couples your write availability to your slowest replica. With one sync follower, a write needs two nodes up; with the leader plus N sync followers, a write needs all N+1 up — so adding replicas for durability would lower availability, the opposite of what you wanted. This is the tension the next two lessons formalize as CAP: you cannot have low-latency writes, strong read consistency, and tolerance of replica failure all at once. Semi-sync (one sync, rest async) is the pragmatic middle — it bounds the durability risk to “lose the leader and its one in-flight sync ack” without holding every write hostage to every replica.
Failover, and the writes that vanish
When the leader dies, the system must fail over: promote a follower to be the new leader. This is where asynchronous replication can silently lose data. Suppose the leader acknowledged writes 98, 99, and 100, but only shipped up to 97 to the followers before it crashed. The promoted follower’s latest write is 97. Writes 98–100 were acknowledged to clients — the users saw success — and now they simply do not exist. Worse, if the old leader comes back and rejoins as a follower, those writes may be on it but conflict with new writes the cluster accepted in the meantime; the usual resolution is to discard them.
This is the failure mode to burn into memory: with async replication, an acknowledged write is not a durable write. The client was told “saved” before the data existed anywhere but the leader, and a crash in that window erases it without a trace. GitHub’s well-known 2012 outage and many others trace to exactly this — a failover that lost or resurrected writes. The mitigations are the durability levers above: at least one synchronous replica so an ack means “on two nodes,” and careful failover tooling that refuses to promote a follower too far behind or fences the old leader so it can’t accept writes after it’s replaced.
▸Common mistake
A subtle and dangerous failover bug is split-brain: the network partitions, the cluster wrongly concludes the leader is dead and promotes a follower, but the old leader is alive and still taking writes on the other side of the partition. Now there are two leaders, both accepting conflicting writes, and when the partition heals you have two divergent histories to reconcile — often by throwing one away. The defenses are fencing (a promoted node forces the old leader to step down, or STONITH — “shoot the other node in the head”) and quorum-based promotion (a follower may only become leader if a majority agrees), which the distributed track’s leader-election and Raft lessons develop in full. The composition-level point: never let “promote a new leader” happen without a mechanism that guarantees the old one can no longer write.
A payment service stores transaction records. Requirements: an acknowledged write must survive a single-node crash (durability), write latency must stay under 20 ms at p99, and the service must tolerate one replica being down without blocking writes. Which replication mode fits?
Multi-leader and leaderless
Two other topologies trade away the single leader:
Multi-leader allows writes on more than one node (e.g. one leader per region, or an offline-capable client). It buys write locality and offline writes, but the price is write conflicts: two leaders accept conflicting updates to the same record, and you must resolve them — last-write-wins (which silently drops data), version vectors, or application-defined merges (CRDTs). Multi-leader doesn’t remove the consistency problem; it moves it from “stale reads” to “conflicting writes.”
Leaderless (the Dynamo model) has no leader at all: the client writes to several nodes and reads from several nodes, and correctness comes from quorums. With N replicas, require W nodes to ack a write and R nodes to answer a read; if R + W > N, the read set and write set must overlap in at least one node, so a read is guaranteed to see the latest acknowledged write. Tune W = N for read-heavy durability or W = 1 for write availability; the overlap inequality is the dial. This is the bridge to the next two lessons — leaderless quorums are how Dynamo-style stores stay available during partitions, at the cost of eventual rather than strong consistency, and the quorum mechanics get a full treatment in the distributed track’s quorum unit.
A user updates their profile, sees 'Saved', and on the very next page load the old value is shown. No errors anywhere. What is the cause and the right fix?
Your async-replicated database acknowledges write #100 to the client, then the leader crashes before shipping #98–100 to followers. A follower at #97 is promoted. What happened to #98–100?
In a leaderless store with N replicas, a read is guaranteed to see the latest acknowledged write whenever the read quorum R and write quorum W satisfy R + W _______ N — because then the set of nodes you wrote to and the set you read from must share at least one node.
- 01Contrast synchronous and asynchronous leader-follower replication, and what semi-sync buys.
- 02What is read-your-writes, why does replication lag break it, and how do you fix it?
- 03How does asynchronous replication silently lose acknowledged writes on failover, and what defends against it?
Replication keeps the same data on multiple nodes to scale reads, place data near users, and survive node death — but it converts a consistency problem into a design choice. In leader-follower, all writes hit one leader that streams its log to followers; the write is synchronous (durable on ≥2 nodes but blocked by the slowest replica) or asynchronous (fast and available but the leader can ack a write that exists nowhere else yet), with semi-synchronous as the usual middle. The lag between leader and follower breaks read-your-writes — the hook’s vanishing-address bug — fixed by routing a user’s own recent reads to a current node. The headline failure mode: with async replication an acknowledged write is not a durable write, so a crash before the write ships, then a failover, loses data the client saw succeed; defend with a synchronous replica and with fencing/quorum promotion to avoid split-brain (two leaders). Multi-leader trades stale reads for write conflicts; leaderless (Dynamo) uses quorums where R + W > N guarantees the read and write sets overlap. The quorum mechanics and leader election here get their full derivation in the distributed track — this lesson stays at the composition altitude of which disagreement you can tolerate. Now when you see a user report “my change disappeared,” your first question is whether the read hit a lagging follower — and you’ll know exactly which lever to pull.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.