Observability
Logs and volume: the cost model of structured logging
Crux How structured logs work, why ingestion bytes dominate the bill, and how level discipline and tiered retention control the cost.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at middle altitude — in the skyA team’s logging bill quadrupled in two months while traffic is flat. One refactor quietly changed a retry count from 3 to 20 — and with it, an INFO log line emitted per attempt. Sixty GB per day became 600 GB per day from one service.
What a structured log is
A log line is an event record: a timestamp plus a payload that describes what happened. Production observability in 2026 is structured-or-nothing: a JSON event with consistent keys is the only form that supports filter-aggregate queries at scale.
Mandatory fields per log line:
timestamp(ISO-8601 UTC)level(debug/info/warn/error)service.nametrace_id(the join key to traces)span_idmessage- Plus event-specific fields (
user_id,order_id,status_code, etc.)
The join key trace_id is what makes it possible to navigate from a metric spike to a log line to a trace with a single click. Without it, you have three disconnected dashboards and no narrative.
The storage model
The pipeline is append-only ingest with indexed search:
- Application emits JSON to stdout or a local agent (Fluent Bit, Vector, OTel Collector).
- Agent buffers and ships to a log backend (Loki, ClickHouse, Elasticsearch, Datadog, Splunk).
- Backend indexes a subset of fields for fast filtering; stores the rest for raw scan.
Logs preserve everything you wrote — high cardinality is free at write time but expensive at query time, since the backend must scan or index every field. Unlike metrics, there is no cardinality explosion from adding user_id to a log line; the cost is paid in ingestion bytes and query scan time, not in series-count memory.
| Log level | Meaning | Production default |
|---|---|---|
| debug | Verbose trace of internal state | Off |
| info | State-change events (request completed, order created) | On, but sample high-volume paths at 1% |
| warn | Recoverable failure (retry succeeded after N attempts) | 100% |
| error | Unrecoverable, actionable failure | 100% |
Ingestion volume: the cost math
A modest service at 1 000 req/s emitting 1 KB of structured JSON per request produces:
- 1 MB/s = 86 GB/day = 2.6 TB/month per service
Multiply by 50 services in a typical fleet and the bill at $0.10/GB ingest plus indexed-event cost reaches tens of thousands per month before retention costs.
Standard cost-control patterns:
- Log-level discipline. DEBUG off in production. INFO for state-change events only. WARN for recoverable failures. ERROR for actionable failures.
- Sampling on success paths. 1-in-100 for high-volume successful INFO events; 100% for errors and warns.
- Tiered retention. 15-day hot for queries; archive to cheap object storage for compliance. Most log queries happen in the first 3 days.
- Redirect chatty tiers. Datadog Flex Logs and Splunk tiered indexes let you route high-volume/low-value events out of the expensive indexed tier into a cheaper scan tier.
Why this works
The retry-log antipattern is the most common cause of surprise log bills: a refactor raises retries from 3 to 20, each attempt emits an INFO line, and the service that was responsible for 5 GB/day suddenly produces 60 GB/day. The fix is to emit a single structured log at the end of the retry sequence — {attempts: 20, final_outcome: "timeout"} — instead of one line per attempt. A metric counter payment_retries_total{outcome} captures the volume cheaply.
Order the steps
Completed
A logging bill quadrupled while traffic is flat. Order the diagnosis steps:
- 1 Open vendor dashboard — find ingest GB/day per service
- 2 Identify the service whose volume jumped (e.g., from 5 GB/day to 60 GB/day)
- 3 Aggregate log volume by log_pattern inside that service
- 4 Find the chatty log line (e.g., one per retry attempt)
- 5 Assess the correct log level (retry attempt = DEBUG or one WARN on final failure)
- 6 Apply short-term mitigation: route to sampled tier or raise level to WARN
- 7 Durable fix: emit one summary log at end of retry sequence + add a metric counter
Quiz
Completed
A service emits an INFO log line for every retry attempt. After a refactor raises retries from 3 to 20, the logging bill 4x in two months. What is the durable fix?
Heads-up Plaintext logs are harder to query and filter; the real problem is volume, not JSON encoding overhead.
Heads-up Reducing retries changes application behaviour; the real fix is log discipline, not business logic.
Heads-up That removes a critical diagnostic signal for failures.
Quiz
Completed
Why does adding user_id to a log line not cause the same explosion as adding it as a metric label?
Heads-up Compression helps but is not the reason; the architectural difference is write-path vs series-count.
Heads-up Logs support arbitrary fields including high-cardinality ones; that is precisely their strength over metrics.
Heads-up No automatic deduplication happens; each event is stored as a separate record.
- Typical log volume per service at 1k req/s with 1 KB per request
- ~86 GB / day
- Datadog log ingest
- ~$0.10 / GB
- Datadog indexed log events
- ~$1.27 / million
- Hot retention typical (query tier)
- 15 days
- Recommended success-path log sampling
- 1% INFO, 100% WARN+ERROR
- Retry-antipattern volume multiplier (3→20 retries)
- ~6-7x bill increase
- 01What are the mandatory fields in a production-quality structured log line, and why is trace_id particularly important?
- 02A service produces 86 GB/day of logs. Name three controls to reduce this without losing actionable data.
- 03Why is structured (JSON) logging required for filter-aggregate queries at scale, while plaintext is not?
Recap
A structured log is a timestamped JSON event with consistent keys; it is the only form that supports filter-aggregate queries at production scale. The backend pipeline is append-only ingest with indexed search, storing every field you write. Unlike metrics, logs have no cardinality explosion from high-cardinality fields — but they pay in ingestion bytes: 86 GB/day per service is typical at 1 000 req/s. Level discipline (DEBUG off, INFO sampled, WARN/ERROR 100%) and tiered retention control the bill. The mandatory trace_id field is the join key that connects log lines to traces, enabling single-click pivots from a log search to a full span waterfall in the tracing backend.
Connected lessons
deepens into
- Observability 2.0: wide events and the cost shiftsenior
- Failure modes and engineering practice: cardinality budgets, PII, and samplingsenior
- Three pillars: build a navigable observability surfacesenior
- Three pillars: multiple-choice reviewsenior
- Three pillars: code and config readingsenior
- Three pillars: free-recall reviewsenior
appears again in167
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What an index is and how it speeds up queriesjunior
- The leading-column rule and composite index designmiddle
- Partial, expression, and covering indexesmiddle
- Index types: GIN, GiST, BRIN, Hash, Bloom, and HOT updatesmiddle
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- Index design exercise: full-text search strategysenior
- EXPLAIN and execution plans: what the planner decides and whyjunior
- Scan types: Seq, Index, Bitmap, Index-Onlymiddle
- Join algorithms and the row-estimate cascademiddle
- pg_statistic, ANALYZE, and production observabilitymiddle
- Extended statistics: fixing correlated-column estimate failuressenior
- Plan cache, cost-constant tuning, and planner internalssenior
- Production failure modes and plan stabilitysenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Migration failure taxonomy and production disciplinesenior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- Why profile first: measure where time actually goesjunior
- Amdahl''''s law and self-time: the ceiling on every speedup you can shipmiddle
- The measurement loop: microbench, macrobench, prod profile, observer effectmiddle
- Reading flame graphs: shapes, per-language profilers, and the 60-second scanmiddle
- Statistical baselines: why one run is not a measurementmiddle
- Profiler history and microbenchmark pitfalls: Knuth to GWPsenior
- Hardware counters, cold-start profiles, and profile securitysenior
- Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patternssenior
- What makes a hot path: symptom vs causejunior
- Five shapes of hotspot: CPU, alloc, cache, lock, syscallmiddle
- Reading parent and child chains: where to apply the fixmiddle
- JIT deopt, the fix-and-verify loop, and PR-time profilingmiddle
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Memory hierarchy: why the same O(N) loop can be 17x slowerjunior
- Row-major vs column-major: access order and the 9x gapjunior
- Branch prediction and branchless codemiddle
- Hardware prefetcher, TLB, and memory-level parallelismsenior
- GC basics: what the runtime taxes you forjunior
- GC algorithms: generational, concurrent, and per-runtimemiddle
- GC tradeoffs: pause, throughput, heap — and object poolingmiddle
- GC tuning: pacing, heap shape, and allocation observabilitymiddle
- GC internals: tri-color invariant, write barriers, and per-runtime deep-divessenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- N+1: one logical operation, many round-tripsjunior
- Fix families: JOIN, IN, preload, and DataLoadermiddle
- Detecting N+1: query logs, APM traces, and CI gatesmiddle
- DataLoader: batching across resolver treesmiddle
- Cross-protocol N+1: HTTP fan-out and Redis MGETmiddle
- N+1 at scale: pool exhaustion, plan changes, and denormalisationsenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- What a bundle actually costs: download, parse, compile, executejunior
- Core Web Vitals: LCP, INP, and CLSmiddle
- Code splitting: route-level, component-level, vendor splittingmiddle
- Tree shaking and compression: removing what you don''''t usemiddle
- Third-party scripts: the silent budget killermiddle
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior