Observability, production failures, and global-scale design

Your payment API starts returning 422 on 10% of POST /charge requests. The symptom appeared after an SDK release. No double charges yet — but the signal is a client bug that, if left unfixed, will cause them. The dashboard needs to surface this before it becomes a production incident.

Minimum viable observability dashboard

Every production idempotency + retry system needs six metrics:

1. idempotency_key_total sliced by outcome:

• new — first-time keys (healthy baseline)
• replay — successful retries (expected under transient failures)
• in_flight (409) — concurrent racing requests
• mismatch (422) — client reusing key with different intent (bug signal)

A rate of mismatch > 0.1% of total POSTs pages on-call. It means clients have a key-generation bug.

2. retry_attempt_total sliced by attempt number (1, 2, 3, 4+): A spike in attempt-3 retries precedes an outage by minutes — the most persistent clients are already paying the cost of an underlying degradation before it becomes user-visible.

3. retry_delay_seconds_bucket — histogram of actual wait times. A bimodal distribution (lots of retries at 0s and at 30s) suggests jitter is not applied.

4. outbox_lag_seconds — how stale is the outbox relay? Alert at lag > 30s. A growing lag means the relay or downstream broker is failing.

5. dead_letter_queue_depth — alert on growth. A growing DLQ is a permanently failing pipeline — poison messages are accumulating.

6. http_request_outcome sliced by status class — 2xx / 4xx / 5xx / timeout. The 4xx / 5xx split distinguishes client bugs from server failures.

Production failure stories

Stripe 2017: switching the idempotency cache from one Redis cluster to another briefly lost in-flight keys. A handful of double charges resulted. Stripe added Postgres as the permanent authoritative backing store — Redis is now only the hot-path cache.

Knight Capital 2012: an absent feature flag plus aggressive retries against an exchange feed turned a deployment glitch into $440M in unintended trades in 45 minutes. Root cause: at-least-once delivery without idempotent consumers.

AWS S3 September 2017: an internal subsystem returned 500s during an outage. Aggressive retry policies without jitter in clients amplified the failure across the region until AWS published guidance requiring jitter for all SDK retries.

GitHub 2018: a database failover lost the lease on an outbox-relay process. Events accumulated in the outbox for 2 minutes, then all flushed at once — a self-inflicted thundering herd on the consumer side. Mitigation: rate-limit the relay’s publish rate during catch-up (e.g., 500 events/s) to flatten the spike.

Every story has the same root: at-least-once delivery without idempotent consumers, OR retries without jitter.

Cross-protocol: HTTP, gRPC, Kafka

The idempotency token concept is universal across protocols:

Kafka idempotent producer: enable.idempotence=true assigns a Producer ID and sequence number to each message. The broker rejects duplicates within a session. For cross-partition atomicity: Kafka transactions.

Deployment: middleware, not hand-rolled

Hand-rolling idempotency in every endpoint produces inconsistent fingerprint formulas and incompatible TTLs. Middleware libraries do it once:

• express-idempotency (Node.js)
• django-idempotency-key (Python/Django)
• fastapi-idempotency (Python/FastAPI)
• AspNetCore.Idempotency (.NET)
• Stripe’s idempotent-requests (open-sourced)

At the infrastructure layer: AWS API Gateway has built-in idempotency since 2023 (TTL up to 10 minutes per stage). Envoy/Istio expose retry policy per route with retry budgets shared across the cluster.