Observability
SLI, SLO, and the error budget: reliability by the numbers
A product manager asks: “Can we ship the new checkout flow this week?” Without an SLO, the answer is a feeling. With one, it is arithmetic — the error budget either has headroom or it does not.
What the three terms actually mean
SLI (Service Level Indicator) is a ratio: the fraction of requests (or events) that were “good” out of everything that was attempted. For a request-driven service, “good” typically means: returned a successful status code, within an acceptable latency. The formula is always the same shape — good events / total events — and the result is a number between 0% and 100%.
SLO (Service Level Objective) is the target you commit to for that SLI. If you pick 99.9%, you are saying: at least 99.9% of requests must be good. Everything above the target is surplus. Everything below is a failure.
Error budget is what remains under the target: 1 − SLO. A 99.9% SLO leaves a 0.1% budget of failures you are allowed to produce before the SLO is breached.
| Term | Formula | Example (99.9% SLO, 30 days) |
|---|---|---|
| SLI | good_events / total_events | 999,000 / 1,000,000 = 99.9% |
| SLO | target for the SLI | >= 99.9% of requests must succeed |
| Error budget | 1 − SLO | 0.1% = 1,000 failures per million, or ~43 minutes of downtime |
| Burn rate | actual_error_rate / (1 − SLO) | 1.0% error rate = 10x burn (exhausts budget in 3 days) |
The data-plan metaphor
Think of the error budget like a monthly mobile data plan. A 99.9% SLO over 30 days has the same shape as “10 GB per month”: you start the period with a fixed allowance, you spend it as events happen, and if you run out before the period ends, you slow down — no new feature deploys — until the next billing cycle resets.
A small incident is like a 30-second video: barely a dent. A two-hour outage is like streaming HD all weekend: most of the month’s quota gone. Teams that are heavy spenders ration carefully; light spenders ship freely. The budget is a real resource, not a metaphor in the system — when it goes negative, real consequences follow.
Burn rate: the derived quantity that matters most
Burn rate normalises the error rate against the SLO target:
burn_rate = actual_error_rate / (1 − SLO)
At the 99.9% SLO (budget = 0.1% = 0.001):
- 0.1% error rate → burn rate 1.0: you would exhaust the budget exactly by month-end
- 1.44% error rate → burn rate 14.4: you would exhaust the 30-day budget in about 2 days
A burn rate of 1 means “sustainable.” Above 1 means you are on pace to miss the SLO. The burn rate is what alerts and dashboards show, because it is the same number regardless of the SLO target or traffic volume — 14.4x means the same thing on any service.
A concrete scenario
A SaaS team sets a 99.5% SLO. Over a month they serve 10 million requests; 0.5% = 50,000 errors allowed. A config bug in week one burns 30,000 errors in 20 minutes — 60% of the monthly budget in one incident. The burn-rate alert fires; the postmortem uses the burned budget as the severity metric. The remaining 40% covers the next 20 days only at the baseline error rate — no room for a risky deploy. The team ships the planned feature behind a feature flag instead.
Why this works
Without an SLO, “should we ship?” is a political argument — whoever has more authority wins. With an SLO and an error budget, the answer is arithmetic: the budget either has headroom or it does not. This is the single most leveraged cultural shift the framework creates. It turns engineering reliability into a language product managers, SREs, and executives can all read.
A service has a 99.9% availability SLO over 30 days. What is the error budget?
A service has a 99.9% SLO and is currently running at a 1.44% error rate. What is the burn rate?
Order the steps of building an SLO from scratch:
- 1 Identify the user journey that matters (checkout, search, login)
- 2 Pick an SLI: a measurable good/total ratio (e.g. successful_requests / total_requests)
- 3 Set the SLO: a target percentage for that ratio (e.g. 99.9%)
- 4 Compute the error budget: 1 − SLO, over a rolling window (typically 28 days)
- 5 Instrument multi-window multi-burn-rate alerts on top of the SLO
- 6 Write the error budget policy: what happens if the budget is exhausted
- 7 Review the SLO quarterly: tighten, loosen, or change the SLI based on real user impact
Fill in the blank: the error _______ is the failure allowance you can spend before a deployment freeze is triggered.
- 01In one sentence each, what is an SLI, an SLO, and an error budget?
- 02Why is burn rate more useful than raw error rate for alerting?
- 03What happens when the error budget reaches zero?
An SLI is the measurable ratio — good events over total events — that tells you how the service is performing from the user’s perspective. The SLO sets the target for that ratio: a 99.9% SLO means 99.9% of events must be good. The error budget is the 0.1% of failures you are allowed before the SLO is breached, converted into concrete numbers: at 1 million requests per month, that is 1,000 failures, or roughly 43 minutes of downtime. Burn rate normalises the current error rate against the budget rate — a burn rate of 14.4x means the 30-day budget would be gone in 2 days — making alerts and dashboards comparable across any service. When the budget is exhausted, the error budget policy halts feature deploys until it regenerates.
appears again in175
- Why GraphQL gets N+1junior
- DataLoader mechanics: tick-boundary batchingmiddle
- Batch function contracts: ordering, shapes, errorsmiddle
- Federation and lookahead: batching beyond DataLoadermiddle
- Query complexity defences: depth, cost, persisted queriesmiddle
- Senior GraphQL API: scheduling contract, tenant isolation, observabilitysenior
- Why idempotency: making retries safejunior
- Server-side state machine: four states of an idempotency keymiddle
- Outbox and inbox: effectively-once across the dual-write boundarymiddle
- Concurrency and cache architecture for idempotency at scalesenior
- Observability, production failures, and global-scale designsenior
- The event loop: one thread, three queuesjunior
- Tasks, microtasks, and scheduler.yield()middle
- Microtask starvation, Long Tasks, and LoAFsenior
- Node.js event loop: phases, nextTick, and loop lagsenior
- React, Vue, and INP observability in productionsenior
- The render pipeline: six stages from bytes to pixelsjunior
- Stage costs and the renderer process modelmiddle
- Invalidation, dirty bits, and containmiddle
- Compositor layers: promotion, overlap, and GPU memorymiddle
- DevTools flame strip and the frame lifecyclemiddle
- Layout thrash: forced synchronous layoutsenior
- BeginMainFrame, compositor-driven animations, and GPU memorysenior
- Production observability: LoAF, INP, and the full attack surfacesenior
- What V8 is and why performance varies 100×junior
- V8''''s four-tier JIT pipeline and profile-guided tieringmiddle
- Hidden classes, transition trees, and memory layoutmiddle
- Inline caches, IC states, and deoptimizationmiddle
- Orinoco GC: parallel scavenger, concurrent marking, and write barriersmiddle
- TurboFan''''s speculative engine and the deopt-loop trapsenior
- V8 in production: isolates, pointer compression, and real failuressenior
- Service worker lifecycle and cache strategiesmiddle
- Service worker edge cases: version skew, durability, and navigation trapssenior
- What the reconciler does: render vs commitjunior
- The fiber object and the double-buffer treemiddle
- Render phase purity and commit phase sub-stepsmiddle
- Reconciliation: diffing heuristics and the key trapmiddle
- Priority lanes, time-slicing, and useTransitionmiddle
- Bailout, memoisation, and tearingsenior
- React Profiler, the Compiler, and production observabilitysenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Hydration mismatch: causes, detection, and the determinism rulesenior
- RSC, per-route strategy, and production observabilitysenior
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- CLS: why layout shifts happen and how to stop themmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What is a cache stampede and why it makes things worsejunior
- Lock and single-flight: bounding concurrent rebuildsmiddle
- XFetch: coordination-free probabilistic early expirationmiddle
- Stale-while-revalidate and CDN request coalescingmiddle
- Detecting stampedes and designing TTL for productionmiddle
- Metastable failure, fencing tokens, and production postmortemssenior
- What a relation is: tables, rows, keys, and constraintsjunior
- Constraints, keys, and Postgres data typesmiddle
- Normal forms, denormalization, and why schemas stickmiddle
- JSONB, arrays, and when a side table winsmiddle
- Heap storage, TOAST, and column alignmentsenior
- Schema integrity: deferral, versioning, and production failure modessenior
- Relational vs document, wide-column, graph, and key-valuesenior
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- pg_statistic, ANALYZE, and production observabilitymiddle
- Production failure modes and plan stabilitysenior
- MVCC: why readers and writers never wait for each otherjunior
- Row versions and snapshots: the on-disk mechanicsmiddle
- HOT updates and isolation levels: what you gain and what you paymiddle
- Vacuum and bloat: keeping the storage tax boundedmiddle
- CLOG, XID wraparound, and MultiXact: deep visibility internalssenior
- SSI internals and production autovacuum tuningsenior
- Real-world MVCC failures, deployment patterns, and distributed snapshotssenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- What a schema migration is and why it replaces ad-hoc DDLjunior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Expand-contract: zero-downtime for breaking schema changesmiddle
- Advisory locks, migration tools, and deploy coordinationsenior
- Migration failure taxonomy and production disciplinesenior
- Why sharding exists: the single-Postgres ceilingjunior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Partitioning vs sharding: same word, two different thingsmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Schema-based sharding and multi-tenancy alternativessenior
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Raft roles, terms, and why majority quorums prevent split brainjunior
- How Raft replicates a log entry and decides it is safe to commitmiddle
- Raft leader election: timeouts, voting rules, and the four safety propertiesmiddle
- Raft in the real world: partitions, slow disks, and client routingmiddle
- Raft extensions: pre-vote, learners, snapshots, and linearizable readssenior
- Raft in production: membership changes, Multi-Raft, and observabilitysenior
- Where data fetching happens — and why it decides LCPjunior
- Fetch waterfalls — diagnosis and the Promise.all curemiddle
- React Server Components and Suspense streamingmiddle
- Client-side cache: TanStack Query, SWR, and stale-while-revalidatemiddle
- LCP, prefetch, and race conditions in interactive fetchingmiddle
- Senior internals: RSC payload, caching layers, and production failure modessenior
- The three-way handshakejunior
- Sequence numbers and connection statemiddle
- DNS: what it does and why it existsjunior
- The resolver walk: referrals, record types, and gluemiddle
- TTL, caching, and DNS propagationmiddle
- The 1-RTT handshake: key shares and ECDHEmiddle
- Session resumption and 0-RTTmiddle
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket frame format: opcodes, masking, fragmentationmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Health checks, connection draining, and slow startmiddle
- Session affinity, consistent hashing, and the right fixmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- Connection IDs and network migrationmiddle
- 0-RTT resumption and packet encryptionsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- Cache lines, struct layout, and false sharingmiddle
- SIMD, SoA vs AoS, and memory bandwidthmiddle
- Cache-oblivious algorithms, PGO, and production failuressenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior
- At-most-once, at-least-once, exactly-once: the three delivery contractsjunior
- The three failure legs — where duplicates and losses actually happenmiddle
- Consumer-side dedup: the cheapest path to exactly-once processingmiddle
- Kafka exactly-once semantics: idempotent producer and transactionsmiddle
- SQS visibility timeout, DLQ, and the outbox patternmiddle
- Exactly-once in production: impossibility proof, hybrid patterns, and real incidentssenior
- What OAuth is and why passwords are not the answerjunior
- Authorization code flow with PKCEmiddle
- ID token validation and JWKS cache managementmiddle
- Refresh token rotation and scope-based least privilegemiddle
- Sender-constrained tokens: DPoP and mTLSsenior
- OAuth in production: audience attacks, observability, and real failuressenior