Observability
Log levels and alert routing
An on-call engineer’s Slack is flooded with ERROR alerts. Half are retry attempts that succeeded. The alerts are noise, real errors get buried, and the oncall tunes out. The bug is not the service — it is the level assigned to the log line.
The level ladder
The production ladder has six rungs: TRACE / DEBUG / INFO / WARN / ERROR / FATAL. Each rung is a contract about what action the log line implies.
| Level | Meaning | Production default | Alert routing |
|---|---|---|---|
| TRACE | Very verbose, framework internals | Off (almost never) | None |
| DEBUG | Developer investigation detail | Off (dynamic flag only) | None |
| INFO | State changes on the success path | On (default) | Query during investigation |
| WARN | Recoverable issues, degraded path taken | On | Slack, capacity dashboards |
| ERROR | Unrecoverable, work left undone | On | Page on-call |
| FATAL | Process is about to die | On | Page on-call immediately |
The production default: INFO and above
The 2026 production default is INFO and above. DEBUG is enabled only via a dynamic flag during active investigation — never on by default. TRACE rarely exists outside of framework internals.
INFO represents the success path: state changes that let a human reconstruct what the service did — request_received, request_completed, kafka_message_processed, transaction_committed. Without INFO lines you cannot reconstruct normal behaviour before an error.
WARN is for recoverable issues: retry succeeded after 3 attempts, fallback path taken, deprecation hit, rate limit approached. The system continued; action may be needed eventually.
ERROR is for unrecoverable issues that left work undone: the request returned 5xx, a message went to the dead-letter queue, a write rolled back. Someone should investigate.
FATAL is for “the process is about to die”: assertion failure, out-of-memory imminent. The process typically exits or is restarted immediately after.
Alert routing follows the level ladder
The level contract implies the alert routing:
- ERROR + FATAL feed pages and incident channels. A page is an implicit promise that action is required.
- WARN feeds Slack and capacity dashboards. No immediate action required; trend is worth watching.
- INFO is queried during investigation, never alerted on directly. An alert on INFO volume would fire constantly on normal traffic.
- DEBUG / TRACE are filtered out in production storage entirely.
Why misclassified levels hurt
INFO used where WARN belongs: the issue gets buried in the INFO stream. WARN routing does not fire. The slow degradation is invisible until it becomes an ERROR.
ERROR used where WARN belongs: every retry, every cache miss, every expected timeout fires a page. The on-call tunes out. When a real ERROR arrives, it is buried in the noise. Alert fatigue is one of the most dangerous observability failures — and it is almost always caused by over-classified levels.
DEBUG left on in production: runaway cost. A service emitting DEBUG at 1000 req/s logs internal request details on every call. At pino’s ~140k msg/sec throughput this is still feasible, but the volume — 100x the INFO stream — hits the backend at full ingest cost. Teams have received 10x monthly log bills from a single PR that left a debug flag enabled.
Why this works
Dynamic DEBUG enabling is the production-safe pattern: the DEBUG level is set to OFF by default via an environment variable or a feature flag, and can be toggled at runtime (typically by updating a config map or calling a management endpoint) for a specific service instance during active investigation. After 30-60 minutes the toggle expires or is manually turned off. This pattern lets you get DEBUG detail when you need it and guarantees you never pay for it in steady state.
A retry-attempt loop emits an ERROR on each of its 5 attempts, including the 3 that succeed. What is wrong with this classification?
In production, which log level is the right default to emit for normal request handling (e.g., request_completed with status 200)?
Map each scenario to the correct log level (order from TRACE to FATAL):
- 1 A specific SQL query's bind parameters during debugging
- 2 A checkout request completed successfully (status 200)
- 3 A retry attempt succeeded on attempt 3 of 5
- 4 A request returned 503 because the upstream payment gateway timed out and work is undone
- 5 The application caught an unhandled exception and is about to exit
- 01What is the difference between WARN and ERROR, and why does it matter for alert routing?
- 02Why is leaving DEBUG on in production a cost risk, not just a noise risk?
- 03What is the dynamic DEBUG flag pattern, and why is it preferable to a process restart?
The six-level ladder — TRACE, DEBUG, INFO, WARN, ERROR, FATAL — serves two purposes: volume control (what gets stored) and alert routing (who gets notified). The production default is INFO and above; DEBUG is off by default and enabled only via a dynamic flag during investigation. INFO logs the success path, WARN logs recoverable degradations, ERROR logs unrecoverable failures that left work undone, and FATAL logs imminent process death. Alert routing follows directly: ERROR and FATAL page on-call, WARN feeds dashboards, INFO is queried during investigation. Misclassified levels create two failure modes: over-classification (recoverable issue as ERROR) causes alert fatigue; under-classification (unrecoverable issue as WARN) makes real failures invisible.
- PII redaction and log injectionsenior
- Trace context propagation in logssenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- Structured logging: build a production logging pipelinesenior
- Structured logging: multiple-choice reviewsenior
- Structured logging: code and log readingsenior
- Structured logging: free-recall reviewsenior
appears again in167
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What an index is and how it speeds up queriesjunior
- The leading-column rule and composite index designmiddle
- Partial, expression, and covering indexesmiddle
- Index types: GIN, GiST, BRIN, Hash, Bloom, and HOT updatesmiddle
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- Index design exercise: full-text search strategysenior
- EXPLAIN and execution plans: what the planner decides and whyjunior
- Scan types: Seq, Index, Bitmap, Index-Onlymiddle
- Join algorithms and the row-estimate cascademiddle
- pg_statistic, ANALYZE, and production observabilitymiddle
- Extended statistics: fixing correlated-column estimate failuressenior
- Plan cache, cost-constant tuning, and planner internalssenior
- Production failure modes and plan stabilitysenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Migration failure taxonomy and production disciplinesenior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- Why profile first: measure where time actually goesjunior
- Amdahl''''s law and self-time: the ceiling on every speedup you can shipmiddle
- The measurement loop: microbench, macrobench, prod profile, observer effectmiddle
- Reading flame graphs: shapes, per-language profilers, and the 60-second scanmiddle
- Statistical baselines: why one run is not a measurementmiddle
- Profiler history and microbenchmark pitfalls: Knuth to GWPsenior
- Hardware counters, cold-start profiles, and profile securitysenior
- Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patternssenior
- What makes a hot path: symptom vs causejunior
- Five shapes of hotspot: CPU, alloc, cache, lock, syscallmiddle
- Reading parent and child chains: where to apply the fixmiddle
- JIT deopt, the fix-and-verify loop, and PR-time profilingmiddle
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Memory hierarchy: why the same O(N) loop can be 17x slowerjunior
- Row-major vs column-major: access order and the 9x gapjunior
- Branch prediction and branchless codemiddle
- Hardware prefetcher, TLB, and memory-level parallelismsenior
- GC basics: what the runtime taxes you forjunior
- GC algorithms: generational, concurrent, and per-runtimemiddle
- GC tradeoffs: pause, throughput, heap — and object poolingmiddle
- GC tuning: pacing, heap shape, and allocation observabilitymiddle
- GC internals: tri-color invariant, write barriers, and per-runtime deep-divessenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- N+1: one logical operation, many round-tripsjunior
- Fix families: JOIN, IN, preload, and DataLoadermiddle
- Detecting N+1: query logs, APM traces, and CI gatesmiddle
- DataLoader: batching across resolver treesmiddle
- Cross-protocol N+1: HTTP fan-out and Redis MGETmiddle
- N+1 at scale: pool exhaustion, plan changes, and denormalisationsenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- What a bundle actually costs: download, parse, compile, executejunior
- Core Web Vitals: LCP, INP, and CLSmiddle
- Code splitting: route-level, component-level, vendor splittingmiddle
- Tree shaking and compression: removing what you don''''t usemiddle
- Third-party scripts: the silent budget killermiddle
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior