Iceberg SLIs, composite SLO math, and SLA vs SLO

A checkout service has a 99.9% availability SLO and it’s green. Customers are reporting duplicate charges. The SLO is not wrong — it’s measuring the wrong thing.

SLI selection: the iceberg failures

The SLI that breaks teams is the one that fails to correlate with user pain. The classic mistake: define an availability SLI on 5xx responses only. What this misses:

4xx-due-to-internal-bug: A 422 thrown because of a serialization error in the service returns a client-error code — not counted as a server failure, not counted as an SLO violation. The user’s request failed. The SLO is green.

High-latency-but-success: A request that took 30 seconds but returned 200 OK passes the availability SLI. The user experienced a failure (30s is above any reasonable latency threshold). If there’s no latency SLO, the SLO system misses it entirely.

Correctness errors: A charge confirmed for the user but missing from the backend. A search result that returned 200 with stale data from three hours ago. The HTTP status was fine; the data was wrong. The SLI didn’t measure it.

Missing events: A messaging service that drops 0.1% of notifications silently — no error, no timeout, just no delivery. The SLI on the send endpoint shows 100% success. The user’s message never arrived.

The senior pattern: SLI selection starts with “what is a bad outcome for the user?” then derives the measurement.

For a checkout service, bad outcomes:

• Charge failure: → availability SLI on payment endpoint
• Charge too slow: → latency SLI (99th percentile under 3s)
• Charge confirmed but missing in backend: → correctness SLI (confirmed_charges_matched / confirmed_charges_total, from a background reconciliation)
• Duplicate charge: → uniqueness SLI (distinct charge IDs / total charges attempted)

Multiple SLIs per journey is normal. The SLO platform aggregates them with worst-of joins: the journey SLO is breached if any of the SLIs violates its target.

Composite SLO math

A user journey through N independent services has a ceiling:

journey_slo = ∏(S₁, S₂, ..., Sₙ)

With independent failures, each service must succeed for the journey to succeed. Five services at 99.9% each: 0.999⁵ ≈ 0.995 — a journey ceiling of 99.5%, not 99.9%.

With dependent failures, the ceiling is lower. Shared database, noisy neighbour, regional network event: when a regional dependency degrades, all five services can fail simultaneously. The independence assumption fails, and ∏Sᵢ is an optimistic upper bound. The actual journey availability in a correlated failure event is worse than the math predicts.

Two architectural levers to raise the ceiling:

Parallel hedging: Fire two requests to two replicas, take the first that succeeds. The probability that both fail: (1 - S)². At 99.9% per replica: 0.001² = 0.000001 — journey availability jumps to 99.9999%. Cost: double traffic, double backend load, must handle the “extra” response (cancel it or deduplicate). Suitable for read-heavy paths where doubling traffic is acceptable.

Idempotent retries with backoff: On transient failure, retry with exponential backoff. A single retry converts most transient failures into successes. At 99.9% per attempt, one retry gives: 1 - (0.001)² ≈ 99.9999%. Cost: latency (the retry adds at least one round-trip’s delay). The endpoint must be idempotent — retrying a charge cannot create a duplicate charge. Idempotency keys (a client-generated unique ID per attempt) enforce this.

Turning architecture into SLO arithmetic: “Should we add a retry?” becomes measurable. If the current failure rate on the payment call is 0.05%, one retry reduces it to (0.0005)² ≈ 0.000025% — below any threshold. The budget gain in failure reductions per month can be calculated and compared to the latency cost in P99 terms.

Latency SLOs and the happiness threshold

A latency SLO needs a threshold — how slow is “slow enough to count as a failure”?

Perceptual boundaries from UX research:

• 100ms: perceptual limit for “instant” — below this, the UI feels responsive
• 200ms: upper boundary of perceived immediacy for interactive elements
• 1s: threshold above which users notice a delay
• 3s: threshold above which a meaningful fraction of users abandon a web request
• 10s+: effectively a timeout for interactive use cases

For a request-driven API serving a UI: 200ms is the standard latency SLO threshold. For backend API calls between services: 1s is common. For batch jobs: minutes to hours depending on business context.

Multi-threshold SLOs catch tail-latency hiding behind a single percentile. “99% under 500ms” might pass while a degraded long tail fails “90% under 100ms.” Production-grade pattern: define both thresholds, AND them with worst-of logic. A service passes its latency SLO only if both thresholds hold.

SLA vs SLO: do not confuse them

The correct relationship: SLO_internal > SLA_external (stricter target internally).

Typical pattern:

• Customer SLA: 99.9% availability
• Internal SLO: 99.95% availability
• Buffer: 0.05 percentage points

The buffer absorbs incidents: when the internal SLO burns, the team investigates and fixes before the SLA breach threshold. The error budget policy kicks in at the SLO, not the SLA. Customers never see the internal SLO — they see whether the SLA was met, which it nearly always is because the team responded to the tighter internal target.

Setting SLO == SLA is the rookie trap. Every internal incident becomes a contract-breach risk. The error budget for SLO enforcement overlaps exactly with the error budget for legal exposure. There’s no buffer to absorb normal operational noise (deployments, dependency degradations, maintenance).

The buffer size depends on the service’s typical incident response time and the support structure. 0.05–0.5 percentage points is the common range. For a service with a 4-hour mean time to detect + respond, 0.5pp buffer gives the team room to find and fix before the SLA breaches. For a service with 2-minute automated rollback, 0.05pp buffer may suffice.