Read real availability math, config, and code, then do the arithmetic: an error-budget calculation, a serial-availability composition, a redundancy-sizing check, and a retry config that turns a slow dependency into an outage.
SDSenior◷ 14 min
Level
FoundationsJuniorMiddleSenior
Availability incidents live in the numbers, not the prose: an SLO budget, a chain of dependencies multiplied together, a pool sized to N instead of N+1, a retry policy with no backoff. Read each snippet, do the arithmetic in your head, and pick the answer a senior on-call engineer would commit to.
Practise the loop you run during an availability review or an incident: locate the numbers, apply the nines/error-budget/composition math or the retry-and-timeout rules, and choose the change the arithmetic actually supports — not the one that feels reassuring.
What does error_budget_minutes(0.999) return over a 28-day window, and what does it mean?
Heads-up ~4.3 min is FOUR nines (99.99%) over ~30 days. Three nines (99.9%) over 28 days is 0.001 × 28 × 24 × 60 ≈ 40.3 min. Watch which nine you're computing.
Heads-up That's two nines (99%, i.e. 0.01). A 99.9% SLO uses 0.001, giving ≈40.3 min over 28 days, not 403. You're off by a factor of ten — one whole nine.
Heads-up 99.9% explicitly allows 0.1% to fail. The budget is 0.001 × 28 × 24 × 60 ≈ 40.3 min — the spendable error budget, not zero. Treating it as zero discards the whole point of an SLO.
Snippet 2 — the dependency chain
# a request must pass through all four to succeedchain = {"lb": 0.999, "app": 0.999, "cache": 0.999, "db": 0.999}avail = 1.0for a in chain.values(): avail *= aprint(round(avail, 4)) # what prints, and is it >= the 99.9% SLO?
Quiz
Completed
What does this print, and does the path meet a 99.9% SLO?
Heads-up Serial availability multiplies, it doesn't copy one link's value. 0.999⁴ ≈ 99.6%, below the 99.9% SLO. Each added dependency lowers the path's availability.
Heads-up Adding serial components LOWERS availability, not raises it. The product is 0.999⁴ ≈ 99.6%, not 99.96%. You can't reach four nines from four three-nines links in series.
Heads-up For a serial path the product is order-independent: 0.999⁴ ≈ 99.6% regardless of sequence. The point is it's below the SLO, so the chain needs redundancy or fewer hops.
Snippet 3 — the redundancy config
# autoscaler / capacity configpeak_required_instances: 6 # instances needed to carry peak loaddesired_instances: 6 # what we actually run# note: "we run multiple instances, so the tier is redundant"
Quiz
Completed
The team calls this tier redundant because it runs multiple instances. Using the redundancy lesson, what's wrong and what's the fix?
Heads-up Redundancy means surviving a loss while still meeting PEAK, which needs a spare. At desired = peak = 6, losing one leaves 5 for a 6-instance peak: under capacity. That's N, not N+1.
Heads-up That makes it worse — now you're under peak even with zero failures. Redundancy IS about spare capacity sized to peak: you need N+1 = 7, not fewer than peak.
Heads-up A second LB removes a different SPOF (the balancer) but doesn't fix instance redundancy: 6 instances for a 6-instance peak is still N. You need a 7th instance for N+1 in this tier.
Snippet 4 — the retry policy
# client calling a downstream serviceRETRY_CONFIG = { "max_attempts": 5, "timeout_seconds": 30, # per attempt "backoff": "none", # retry immediately "jitter": False,}# the downstream service has just started returning slow responses
Quiz
Completed
The downstream service slows down. What does this retry policy do to it, and what's the correct configuration?
Heads-up During overload, hammering 5× immediately multiplies load exactly when the dependency can least afford it (a retry storm). Reliability comes from capped, backed-off, jittered retries, not from retrying harder.
Heads-up An overloaded dependency has no spare capacity; the retries are added load on a failing system, accelerating collapse. That's why you need a retry budget plus backoff and jitter.
Heads-up Fewer attempts helps a little, but the 30 s timeout (thread exhaustion) and the no-backoff/no-jitter lockstep retries are the bigger issues. Fix timeouts, add exponential backoff WITH jitter, and a retry budget.
Recall before you leave
01
How do you compute an error budget from an SLO, and what's the common nine-off-by-one error?
02
Why does a four-link 99.9% serial path miss a 99.9% SLO, and what are the fixes?
03
What turns a slow downstream into an outage in a retry config, and what's the safe configuration?
Recap
Every availability decision in this unit reduces to arithmetic you can read straight off the config. The error budget is (1 − SLO) × window — 99.9% over 28 days ≈ 40 minutes — and the classic bug is being off by one nine (0.01 vs 0.001, a 10× error). Path availability is the product of its serial links, so a four-link 99.9% chain is only ~99.6% and misses a 99.9% SLO; you fix it by removing hops or adding redundancy, not by editing prose. Redundancy is N+1, so desired_instances == peak_required is N — not redundant — and losing one at peak tips into the queueing collapse; you provision a spare. And a retry policy with long timeouts and immediate, un-jittered retries turns a slow dependency into a retry storm and an outage — the fix is short bounded timeouts, a retry budget, exponential backoff with jitter, circuit breakers, and idempotent-only retries. The senior habit is the same as the scalability unit: find the numbers, do the math, and choose the fix the arithmetic supports rather than the one that sounds safe.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.