Security
OAuth in production: audience attacks, observability, and real failures
A JWT validator logs a WARN when the aud claim does not contain the expected value, then proceeds to accept the token anyway because “aud contains the caller’s client_id.” The token was issued for a different application. The validator just accepted a token intended for someone else — audience bypass, a real CVE pattern seen at Microsoft Azure AD in November 2024.
Audience attacks
When one authorization server issues tokens for multiple resource servers (API-A and API-B), a validator that accepts any valid aud can be exploited: an attacker who obtains a token for API-A presents it to API-B.
The correct aud check: does the token’s aud claim contain this resource server’s hard-coded identifier? Not “does aud contain any known client_id?” — a logical inversion that bypasses the protection.
RFC 8707 — Resource Indicators: The client specifies a resource parameter in the /token request, naming the target resource server. The IdP mints a token with that audience only. If the client needs tokens for both API-A and API-B, it calls /token twice with different resource values. Inconvenient but correct.
Token Exchange (RFC 8693): A resource server that needs to call a downstream service can present its incoming token and request a new token valid at the downstream, with a narrowed audience. This is “least privilege” for service-to-service hops.
The cnf (confirmation) claim also narrows binding: DPoP sets cnf.jkt to the public key thumbprint. A token with cnf can only be used with proof signed by the corresponding private key — audience binding plus sender binding together.
if token.aud.includes(EXPECTED) { accept }
Token storage by client type
Where tokens live determines their exposure surface:
Browser SPA:
- Access token: JavaScript memory only (closure or React state). Never
localStorage— XSS-readable. - Refresh token:
httpOnly; Secure; SameSite=Strictcookie — inaccessible from JS. - On page reload:
prompt=nonesilent/authorizeround using the refresh token cookie.
Native mobile app:
- Access token: OS keychain (iOS Keychain, Android Keystore). Not process memory.
- Refresh token: OS keychain, never in
SharedPreferencesor unprotected storage.
Server-side app:
- Access token: server session (in-memory or Redis, not the client).
- Refresh token: server session. Client never sees either token.
Machine-to-machine (CI/CD, service):
- Use
client_credentialsgrant, not user-delegated tokens. - Never embed user access tokens in CI pipelines.
Production observability
A minimum OAuth observability dashboard:
| Metric | Alert condition |
|---|---|
refresh_replay_detected_total | Any non-zero value — probable compromise |
id_token_validation_failure_total by reason | Spike on sig/kid → JWKS rotation issue |
token_introspection_latency_p99 | Above 200ms → IdP overloaded |
jwks_cache_hit_ratio | Drop below 90% → cache TTL too short |
dpop_proof_failure_total by reason | Spike on iat_skew → clock drift in mobile clients |
token_request_total by outcome | Spike on invalid_grant → attack or misconfigured client |
refresh_replay_detected_total > 0 is the most actionable alert: it means at least one refresh token was used by two distinct clients — someone stole a token.
Four real-world OAuth failures
Facebook, September 2018. A bug in the “View As” feature exposed 50 million access tokens. Attackers could impersonate any of those users at any Facebook OAuth client (Spotify, Tinder, Instagram). Fix: invalidate 90 million tokens.
Slack, February 2017. A misconfigured OAuth redirect URI in a third-party Slack app let an attacker phish a workspace owner’s authorization code. The attacker escalated to full admin access. Fix: URI validation, required redirect URI exact-match enforcement at the IdP level.
GitHub Enterprise, 2021. A bug in PKCE verification allowed downgrade attacks against older clients. Fixed in a patch release.
Microsoft Azure AD, November 2024. A token-validation cache poisoning vulnerability let a crafted token bypass aud validation for ~30 minutes on cached negative responses. Fixed within hours. Root cause: the negative-response cache did not distinguish “aud not found” from “aud explicitly rejected.”
The pattern: every incident involved one skipped or buggy mandatory check. The industry response to all four was the same — more mandatory checks, shorter TTLs, wider observability.
A team wants to share one access token between two resource servers (API-A and API-B) for convenience. Why is this dangerous?
Why should access tokens never be stored in localStorage in a browser SPA?
Order the steps to diagnose a spike in id_token validation failures:
- 1 Check the failure_reason dimension on id_token_validation_failure_total
- 2 If reason is 'sig' or 'kid-not-found', suspect JWKS key rotation
- 3 Check jwks_cache_hit_ratio — a drop indicates stale cache from rotation
- 4 Confirm by checking if the IdP published a new signing key recently
- 5 Trigger immediate JWKS refresh across all instances
- 6 Reduce JWKS cache TTL to 5–10 minutes and add on-cache-miss refresh as backstop
- 01Explain the audience confusion attack and the correct aud check that prevents it.
- 02Why is refresh_replay_detected_total > 0 a compromise indicator rather than a harmless error?
- 03What observability gap does short access token TTL (5–15 min) address?
Audience validation is a hard-coded identity check, not a membership check — the token’s aud must contain this resource server’s specific identifier, never derived from the request. Token storage follows the client’s threat model: JS memory for SPAs, OS keychain for native, server session for server-side. Production observability must expose refresh_replay_detected_total (compromise signal), JWKS cache hit ratio (rotation readiness), and introspection latency (IdP health). The four major OAuth incidents — Facebook 2018, Slack 2017, GitHub Enterprise 2021, Azure AD 2024 — each followed from one skipped or buggy mandatory check. OAuth security is a complete-set problem: every check must pass, every time.
appears again in202
- Why GraphQL gets N+1junior
- DataLoader mechanics: tick-boundary batchingmiddle
- Batch function contracts: ordering, shapes, errorsmiddle
- Federation and lookahead: batching beyond DataLoadermiddle
- Query complexity defences: depth, cost, persisted queriesmiddle
- Senior GraphQL API: scheduling contract, tenant isolation, observabilitysenior
- Why idempotency: making retries safejunior
- Server-side state machine: four states of an idempotency keymiddle
- Outbox and inbox: effectively-once across the dual-write boundarymiddle
- Concurrency and cache architecture for idempotency at scalesenior
- Observability, production failures, and global-scale designsenior
- The event loop: one thread, three queuesjunior
- Tasks, microtasks, and scheduler.yield()middle
- Microtask starvation, Long Tasks, and LoAFsenior
- Node.js event loop: phases, nextTick, and loop lagsenior
- React, Vue, and INP observability in productionsenior
- The render pipeline: six stages from bytes to pixelsjunior
- Stage costs and the renderer process modelmiddle
- Invalidation, dirty bits, and containmiddle
- Compositor layers: promotion, overlap, and GPU memorymiddle
- DevTools flame strip and the frame lifecyclemiddle
- Layout thrash: forced synchronous layoutsenior
- BeginMainFrame, compositor-driven animations, and GPU memorysenior
- Production observability: LoAF, INP, and the full attack surfacesenior
- What V8 is and why performance varies 100×junior
- V8''''s four-tier JIT pipeline and profile-guided tieringmiddle
- Hidden classes, transition trees, and memory layoutmiddle
- Inline caches, IC states, and deoptimizationmiddle
- Orinoco GC: parallel scavenger, concurrent marking, and write barriersmiddle
- TurboFan''''s speculative engine and the deopt-loop trapsenior
- V8 in production: isolates, pointer compression, and real failuressenior
- What workers are and why they existjunior
- Web worker mechanics: dedicated, shared, and OffscreenCanvasmiddle
- Structured clone and transferablesmiddle
- Service worker lifecycle and cache strategiesmiddle
- SharedArrayBuffer, Atomics, and cross-origin isolationsenior
- Service worker edge cases: version skew, durability, and navigation trapssenior
- Worker pools, Comlink, and production observabilitysenior
- What the reconciler does: render vs commitjunior
- The fiber object and the double-buffer treemiddle
- Render phase purity and commit phase sub-stepsmiddle
- Reconciliation: diffing heuristics and the key trapmiddle
- Priority lanes, time-slicing, and useTransitionmiddle
- Bailout, memoisation, and tearingsenior
- React Profiler, the Compiler, and production observabilitysenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Hydration mismatch: causes, detection, and the determinism rulesenior
- RSC, per-route strategy, and production observabilitysenior
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- CLS: why layout shifts happen and how to stop themmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What is a cache stampede and why it makes things worsejunior
- Lock and single-flight: bounding concurrent rebuildsmiddle
- XFetch: coordination-free probabilistic early expirationmiddle
- Stale-while-revalidate and CDN request coalescingmiddle
- Detecting stampedes and designing TTL for productionmiddle
- Metastable failure, fencing tokens, and production postmortemssenior
- What a relation is: tables, rows, keys, and constraintsjunior
- Constraints, keys, and Postgres data typesmiddle
- Normal forms, denormalization, and why schemas stickmiddle
- JSONB, arrays, and when a side table winsmiddle
- Heap storage, TOAST, and column alignmentsenior
- Schema integrity: deferral, versioning, and production failure modessenior
- Relational vs document, wide-column, graph, and key-valuesenior
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- pg_statistic, ANALYZE, and production observabilitymiddle
- Production failure modes and plan stabilitysenior
- MVCC: why readers and writers never wait for each otherjunior
- Row versions and snapshots: the on-disk mechanicsmiddle
- HOT updates and isolation levels: what you gain and what you paymiddle
- Vacuum and bloat: keeping the storage tax boundedmiddle
- CLOG, XID wraparound, and MultiXact: deep visibility internalssenior
- SSI internals and production autovacuum tuningsenior
- Real-world MVCC failures, deployment patterns, and distributed snapshotssenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- What a schema migration is and why it replaces ad-hoc DDLjunior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Expand-contract: zero-downtime for breaking schema changesmiddle
- Advisory locks, migration tools, and deploy coordinationsenior
- Migration failure taxonomy and production disciplinesenior
- Why sharding exists: the single-Postgres ceilingjunior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Partitioning vs sharding: same word, two different thingsmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Schema-based sharding and multi-tenancy alternativessenior
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Raft roles, terms, and why majority quorums prevent split brainjunior
- How Raft replicates a log entry and decides it is safe to commitmiddle
- Raft leader election: timeouts, voting rules, and the four safety propertiesmiddle
- Raft in the real world: partitions, slow disks, and client routingmiddle
- Raft extensions: pre-vote, learners, snapshots, and linearizable readssenior
- Raft in production: membership changes, Multi-Raft, and observabilitysenior
- Where data fetching happens — and why it decides LCPjunior
- Fetch waterfalls — diagnosis and the Promise.all curemiddle
- React Server Components and Suspense streamingmiddle
- Client-side cache: TanStack Query, SWR, and stale-while-revalidatemiddle
- LCP, prefetch, and race conditions in interactive fetchingmiddle
- Senior internals: RSC payload, caching layers, and production failure modessenior
- The IP envelopejunior
- Reading the IP headermiddle
- The three-way handshakejunior
- Sequence numbers and connection statemiddle
- DNS: what it does and why it existsjunior
- The resolver walk: referrals, record types, and gluemiddle
- TTL, caching, and DNS propagationmiddle
- What TLS does and why it existsjunior
- The 1-RTT handshake: key shares and ECDHEmiddle
- Session resumption and 0-RTTmiddle
- Key schedule, SNI, ALPN, and extensionssenior
- 0-RTT defenses, ECH, hybrid PQ, and production TLSsenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket frame format: opcodes, masking, fragmentationmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Health checks, connection draining, and slow startmiddle
- Session affinity, consistent hashing, and the right fixmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- Connection IDs and network migrationmiddle
- 0-RTT resumption and packet encryptionsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- What the three signals are: logs, metrics, and tracesjunior
- Why structured logs exist: the diary vs the spreadsheetjunior
- The production log schema: fields every line must carrymiddle
- PII redaction and log injectionsenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- What is OpenTelemetry: API, SDK, Collector, OTLPjunior
- OTel signals, Semantic Conventions, and the OTLP wire formatmiddle
- The OTel Collector: receivers, processors, exporters, and deployment patternsmiddle
- Vendor neutrality, eBPF instrumentation, the Operator, and browser/serverless OTelsenior
- Operating the OTel Collector: reliability, version skew, failure modes, and governancesenior
- SLI, SLO, and the error budget: reliability by the numbersjunior
- Error budget policy, latency SLOs, and composite journeysmiddle
- Production SLO failures, self-observability, security, and the big picturesenior
- What is trace propagation and why broken propagation is worse than nonejunior
- traceparent and tracestate: the W3C header format in fullmiddle
- Baggage and async boundaries: carrying context across queues and callbacksmiddle
- Async context per language, service mesh, B3 migration, and securitysenior
- Production propagation failures, span links, and platform designsenior
- The debugging funnel: SLO → RED → trace → profilejunior
- OTel architecture: one SDK, four signals, one wire formatmiddle
- The incident loop: from pager to postmortem to preventionmiddle
- Scale, security, and the ROI of observable systemssenior
- Cache lines, struct layout, and false sharingmiddle
- SIMD, SoA vs AoS, and memory bandwidthmiddle
- Cache-oblivious algorithms, PGO, and production failuressenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior
- At-most-once, at-least-once, exactly-once: the three delivery contractsjunior
- The three failure legs — where duplicates and losses actually happenmiddle
- Consumer-side dedup: the cheapest path to exactly-once processingmiddle
- Kafka exactly-once semantics: idempotent producer and transactionsmiddle
- SQS visibility timeout, DLQ, and the outbox patternmiddle
- Exactly-once in production: impossibility proof, hybrid patterns, and real incidentssenior