Failover & fault tolerance
Failover moves traffic off a dead component onto a healthy one. Fault tolerance survives a fault with no visible interruption at all. Get detection, split-brain, and the retry/timeout interaction wrong and your failover causes the outage it was meant to prevent.
A database cluster had a standby ready and automatic failover configured — textbook resilience. Then the network between the primary and standby hiccuped for eight seconds. The standby couldn’t reach the primary, declared it dead, and promoted itself. But the primary wasn’t dead; it was fine, and still accepting writes. For those eight seconds the system had two primaries, both taking writes, diverging. When the network healed, the two write histories conflicted and an engineer spent the night hand-reconciling orders. The redundancy worked. The failover is what caused the outage — because the one thing it couldn’t tell was the difference between “the primary is dead” and “I can’t reach the primary.”
Two words that aren’t the same thing
People say “fault tolerant” when they mean “has failover,” and the gap between them is where expectations break.
- Failover is a reactive sequence: a component dies, something notices, and traffic is moved to a healthy replacement. There is a window — detection plus switchover — during which requests fail or hang. Failover gives you high availability: the system recovers quickly, but not instantly. Active-passive database promotion is failover.
- Fault tolerance is masking a fault so there is no visible interruption at all. The faulty component’s work is absorbed by others with zero gap, because redundancy was already live and serving. Active-active behind a load balancer is fault tolerant for the loss of one node — the survivors were already taking traffic, so losing one is invisible to users.
The practical distinction: a fault-tolerant system has no recovery window for the faults it tolerates; a highly-available system with failover has a short one. True fault tolerance is more expensive (you run and pay for the redundant capacity live, continuously), so most systems are highly available via failover for most components and fault tolerant only where a gap is unacceptable.
The mechanics: detect → promote → redirect → fail back
A failover is a four-step pipeline, and each step has a failure mode:
- Detect. Something must notice the component is gone. Two common mechanisms: health checks (a watcher periodically probes “are you ok?” and expects a healthy response) and heartbeats (the component periodically announces “I’m alive” and silence means death). Detection is a tradeoff: probe too rarely and you’re slow to react; probe too aggressively and a momentary blip looks like death and you fail over needlessly.
- Promote. A standby is elevated to take over — a replica becomes the new primary, a passive node goes active. For stateful systems this is the dangerous step: you must be sure the old one is really gone.
- Redirect. Traffic is pointed at the new target — update DNS, reconfigure the load balancer, flip a virtual IP. This step is bounded by how fast the redirect propagates (DNS TTLs, connection draining).
- Fail back. When the original recovers, you decide whether to return to it (fail back) or leave the promoted node as the new primary. Failing back is itself a controlled failover and a common place to cause a second outage by doing it carelessly under load.
Together these four steps mean every part of your infrastructure that matters — DNS TTLs, health-check intervals, standby warmup time — needs to be engineered before an incident, not improvised during one. Skip the fencing step and you risk split-brain; skip the redirect step and traffic never finds the new primary.
Split-brain: the failure failover causes
The hook’s outage has a name: split-brain. It happens when a failover promotes a new primary while the old one is still alive — usually because a network partition made them unable to see each other, and each side concluded the other was dead. Now two nodes both believe they’re the primary, both accept writes, and the data diverges. When the partition heals you have two conflicting histories and no automatic way to merge them. Split-brain is uniquely nasty because it doesn’t lose availability — both halves are up and serving — it loses correctness, which is often worse and harder to detect.
The root cause is the fundamental ambiguity detection can never fully resolve: from the standby’s point of view, “the primary crashed” and “I lost the network to the primary” look identical — silence either way. The defenses all reduce to making sure at most one node can win:
- Quorum / consensus. Require a majority of nodes to agree before promoting. A partition that leaves the old primary in the minority means it can’t keep acting as primary, and only the majority side promotes. This is why consensus systems (Raft, ZooKeeper, etcd) need an odd number of nodes — a majority must be unambiguous.
- Fencing (STONITH — “shoot the other node in the head”). Before promoting, forcibly cut off the old primary — revoke its storage lease, kill its power, block it at the network — so even if it’s alive it cannot keep writing. You don’t ask it to stand down; you make standing down physically enforced.
▸Why this works
Why can’t you just make detection smarter — longer timeouts, more probes — and avoid the whole problem? Because no timeout resolves the ambiguity; it only trades one bad outcome for another. A short detection timeout fails over fast but mistakes a transient blip for death and risks split-brain (and needless churn). A long timeout avoids false positives but means a real death leaves you down for the whole timeout. There is no setting that’s both fast and safe, because “no response” genuinely cannot be distinguished from “slow response” or “dead” by waiting. That’s why the durable fix isn’t better detection — it’s a mechanism that makes a wrong detection harmless: quorum so the minority can’t promote, or fencing so a wrongly-declared-dead node can’t keep writing. You design for detection to be wrong, not to be perfect.
Graceful degradation: failing partway instead of all the way
Fault tolerance isn’t always all-or-nothing. Graceful degradation is designing the system to shed non-essential functionality under failure instead of going fully dark. If the recommendations service is down, show a generic popular list instead of a 500. If the personalization cache is unreachable, serve the page without personalization. If writes are failing, stay up in read-only mode. The user gets a degraded but working experience, and the failure is contained to the feature that broke. This is the practical complement to redundancy: redundancy keeps the critical path up, graceful degradation makes the non-critical paths fail softly, so one dependency’s death is a missing feature, not a blank page.
The retry/timeout/failover interaction — where failover turns into outage
This is the senior-level trap, and it’s why AWS dedicates a whole Builders’ Library article to it. Failover, timeouts, and retries interact, and misconfigured they amplify a small fault into a cascading one:
- Timeouts must be set deliberately. Too long, and a single slow dependency holds threads hostage until the whole pool is exhausted and you’ve turned one slow backend into a full outage. Too short, and you abandon requests that would have succeeded, manufacturing failures.
- Retries help with transient blips but are dangerous during overload. When a dependency is struggling, naive retries multiply the load on it exactly when it can least afford it — a feedback loop AWS calls a retry storm that drives brownout into outage. The defenses: a retry budget (cap retries as a fraction of traffic, not per-request), exponential backoff (wait longer between attempts), and jitter (randomize the backoff so all clients don’t retry in lockstep and create synchronized spikes). And retries belong only on idempotent operations — retrying a non-idempotent write can double-charge a customer.
- During failover, all of this compounds: the moment the primary dies, every in-flight request times out at once, every client retries at once, and that synchronized surge lands on a standby that’s still warming up — so the failover event itself can knock over the thing you just promoted. AWS’s prescription (bounded timeouts, capped retries, exponential backoff with jitter, circuit breakers) exists specifically so the recovery doesn’t become the next incident.
▸More practice
The two numbers that summarize all of this are MTBF and MTTR. MTBF (mean time between failures) is how often things break; MTTR (mean time to recovery) is how long you’re down when they do. Availability ≈ MTBF / (MTBF + MTTR), so you can raise availability two ways: fail less often (raise MTBF) or recover faster (lower MTTR). The senior insight is that lowering MTTR is usually the better investment. Failures are inevitable and chasing higher MTBF hits diminishing returns, but fast, automated, well-rehearsed recovery — good detection, tested failover, fencing, graceful degradation, runbooks — shrinks every incident. A system that fails twice as often but recovers ten times faster is far more available. This is why mature teams obsess over MTTR: practiced recovery, not the fantasy of never failing.
A network partition isolates the primary from its standby. The standby can't reach the primary, declares it dead, and promotes itself — but the primary is alive and still taking writes. What is this, and what prevents it?
A downstream service slows down. Clients have a long timeout and retry up to 3× with no backoff. What is the likely outcome, and the correct fix?
Availability ≈ MTBF / (MTBF + MTTR), so you can fail less often or recover faster. Mature teams invest most in lowering _______ — the mean time to recovery — because failures are inevitable but fast, rehearsed recovery shrinks every incident.
- 01Distinguish failover from fault tolerance, and name the failover steps.
- 02What is split-brain, why can't better detection prevent it, and what does?
- 03How do retries, timeouts, and failover interact, and how do you keep recovery from becoming the next outage?
Failover and fault tolerance are not synonyms: failover is the reactive detect → fence → promote → redirect → fail back sequence that gives high availability with a short recovery window, while fault tolerance masks the fault with no visible interruption (live redundant capacity, like active-active) at higher cost. The hard part is detection, because “the node crashed” and “I can’t reach the node” are indistinguishable by waiting — and a failover that promotes a new primary while the old one is alive causes split-brain, two primaries taking writes and diverging with no automatic merge. You can’t fix that with smarter timeouts (they only trade false positives against slow recovery); you fix it by making a wrong call harmless — quorum so only the majority promotes, or fencing/STONITH so the old primary can’t keep writing. Graceful degradation shrinks failures further by shedding non-essential features (popular-items list, read-only mode) instead of going dark. The senior trap is the retry/timeout/failover interaction: long timeouts exhaust pools, naive retries during overload become a retry storm, and at failover every client times out and retries in lockstep onto a warming standby — so you bound timeouts and use capped retries with exponential backoff and jitter (only on idempotent ops) so recovery isn’t the next incident. Finally, availability ≈ MTBF/(MTBF+MTTR): since failures are inevitable, investing in fast, automated, rehearsed recovery (lower MTTR) usually beats chasing an ever-higher MTBF — exactly the discipline the hook’s team lacked when their failover, not the fault, caused the outage. Now when you see a failover event in a postmortem, check which step went wrong: was it detection that misfired, a missing fence that caused split-brain, or retries that turned a hiccup into a stampede?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.