Availability: an SLO and failover plan from scratch
Hands-on project: take a product brief, derive its SLIs/SLOs and error budget, hunt every single point of failure and design the redundancy and failover to remove them, then prove your failover and degradation work with a small chaos test that injects failures.
Reading that you should define SLOs and remove single points of failure is not the same as producing an availability design a staff engineer would sign off on. Take a real-shaped product brief, write its SLIs and SLOs with an error budget, map and eliminate every SPOF with the right redundancy and failover, and then prove the two riskiest claims — that failover actually recovers and that the system degrades instead of going dark — with a small chaos test you write yourself.
This project makes the whole unit operational: you’ll choose user-centric SLIs and set SLOs under an SLA, derive the error budget, hunt SPOFs and design redundancy and failover that hold under correlated failure, and then measure — inject failures and confirm the system recovers within your target and degrades gracefully instead of failing whole.
Produce a complete availability design for a chosen product (e.g. a URL shortener, a payments API, or a chat backend): define its SLIs, SLOs, and error budget; map and eliminate every single point of failure with appropriately-sized redundancy and a failover plan; then build a small chaos test that injects failures and empirically validates that failover recovers within your target and that the system degrades gracefully — closing the loop between the design and the failures it claims to survive.
- An SLI/SLO/error-budget sheet: 2-3 user-centric SLIs with where each is measured, an SLO per SLI tighter than the SLA, the computed error budget per window, and the policy for half-spent and exhausted budget — readable in two minutes.
- A SPOF table that covers the whole architecture (including shared infra and non-machine SPOFs), each row with the removal: redundancy level (N+1/N+2), failure-domain placement, and active-active vs active-passive with the tradeoff stated.
- A failover plan naming the detection mechanism and its timeout tradeoff, the promote/redirect steps, the split-brain defense (quorum or fencing), and the timeout/retry-budget/backoff-with-jitter settings that keep recovery from becoming a retry storm.
- A working chaos test whose measured recovery time after killing the active node is reported against the failover target, plus the error-budget burn the simulated incident consumed.
- A demonstration of graceful degradation (a forced-down dependency yields a degraded-but-working response) and of retry safety (the same failure with vs without backoff-and-jitter, showing the storm is averted).
- A short write-up: one paragraph on which SPOF was least obvious and how you found it, and one on where the chaos test's measured recovery differed from the design's assumption and why.
- Add a correlated-failure test: place both replicas in the same simulated failure domain, take the domain down, and show the 'redundant' pair fails together; then re-place them in independent domains and show it survives — quantifying the difference in availability.
- Add a split-brain demonstration: partition a primary from its standby without fencing/quorum and show the divergence; then enable fencing or quorum and show the wrong 'dead' call becomes harmless.
- Model the nines: from your measured MTBF (injected failure rate) and MTTR (measured recovery), compute availability ≈ MTBF/(MTBF+MTTR), and show which investment — failing less often vs recovering faster — moves your achievable SLO more.
- Add an open-loop retry storm reproduction: hold the dependency slow, run clients with naive immediate retries vs a retry budget with backoff-and-jitter, and plot the load amplification each produces on the struggling dependency.
- 01What does the availability design produce before any chaos test, and how does each piece connect?
- 02How do you validate failover and graceful degradation in a chaos test, and what do you measure?
- 03Why design SLOs and SPOF removal first, then chaos-test, rather than just chaos-testing?
This project turns the availability unit into a repeatable workflow: take a product brief, define user-centric SLIs (good/valid ratios measured close to the user), set SLOs tighter than the SLA, and compute the error budget with a spend policy. Then hunt every SPOF with the thought experiment over every box and line — including shared infrastructure and non-machine SPOFs — and remove each with redundancy sized to N+1 against peak, placed in independent failure domains, choosing active-active or active-passive with the tradeoff stated. Write the failover plan (detection and its timeout tradeoff, promote/redirect, a split-brain defense via quorum or fencing, and storm-safe timeouts/retries/backoff-with-jitter) and the graceful-degradation behaviors so a dependency’s death is a missing feature, not a blank page. Then close the loop with a chaos test: kill the active node and measure recovery against the target, force a dependency down and confirm degradation, and reproduce the retry storm with and without backoff-and-jitter. Doing the design first and the chaos test second is the whole discipline — the paper design says what should survive, the test confirms it does. An engineer who has built this once designs for failure deliberately instead of discovering their SPOFs and their split-brain during a 3 a.m. incident.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.