Observability
Observability 2.0: wide events and the cost shift
A 40-service team spends two engineer-weeks per quarter on cardinality budgets and three hours per incident on cross-pillar pivots. The engineering cost of three separate observability backends now exceeds the SaaS bill of a unified one. The pillars were not wrong — they were pre-2020 cost-optimal.
The 2.0 critique
Charity Majors (Honeycomb co-founder, the engineer who put “observability” on the map for distributed systems in 2017) argues in her 2024–2025 blog series: the three pillars are not a natural taxonomy — they are an artifact of late-2000s storage limits.
- Metrics existed because you could not afford to keep every event.
- Logs existed because you could not aggregate them at query time.
- Traces existed because neither of the other two carried cross-service causality.
With columnar storage engines, cheap object storage ($0.023/GB on S3), and modern query planners, all three collapse back into one shape: wide structured events — one record per unit of work, carrying every field including high-cardinality ones (user_id, build_sha, feature_flag_state, region), stored in a single columnar engine, queried with arbitrary slicing.
Honeycomb’s product is this. ClickHouse-based stacks (Signoz, Sentry’s new backend, Cloudflare’s internal stack) are this. The Honeycomb post “OpenTelemetry Is Not Three Pillars” explicitly recasts OTel signals as “ways to ship and store a single underlying telemetry model.”
What changes at the engineering level
In a 1.0 stack each request emits to three places:
- Counter increments → metric SDK
- Log lines → log SDK
- Spans → trace SDK
Each signal is sampled or aggregated differently, fields are denormalised, and the join across them depends on shared join keys being plumbed consistently. Three billing meters, three operational surfaces.
In a 2.0 stack each request emits one wide event per service boundary — a JSON record with all attributes (user_id, route, status, duration, build_sha, feature_flags, trace_id) — and the backend computes metrics by GROUP BY at query time, retrieves logs by filtering, reconstructs traces by joining on trace_id. The columnar engine makes all three queries fast over the same data. Cardinality is no longer a cost driver in 2.0 — that is the headline shift.
| Property | Observability 1.0 | Observability 2.0 |
|---|---|---|
| Storage shape | Three separate backends | One columnar store |
| Billing | series-month + GB-ingest + span-month | ~traffic-month + dimensionality |
| Cardinality as cost | Yes — OOM risk on TSDB | No — query-time GROUP BY |
| Cross-signal pivot | Join keys + exemplars required | Same data, arbitrary slice |
| Backend options | Many, open-source paths | Fewer, often SaaS-only |
The migration decision
The 2.0 architecture is not universally superior. The senior question is: “is the cost basis of 2.0 viable for my workload?”
2.0 wins when:
- Cardinality budget overruns cost more engineer-hours than the bill saves
- Cross-pillar pivot frustration bottlenecks incident response
- Traffic volume and architectural sprawl drive the cost, not storage shape
1.0 wins when:
- Very long retention (5+ years) for low-cardinality metrics — pre-aggregation is hard to beat on cold storage
- Strong open-source preference, avoiding SaaS lock-in
- Budget is dominated by metrics, not log or trace volume
Many shops run both: 2.0 backend for incident response and ad-hoc questions, 1.0 metrics tier (Prometheus + remote storage) for 5-year SLO trend dashboards and regulatory reports.
The migration path is dual-write, not rip-and-replace: point OTel exporters at both backends for 60–90 days, build 2.0 dashboards alongside 1.0 ones, decommission 1.0 only after every team’s on-call run-books have moved.
The vendor-lock-in question
1.0 instrumentation historically coupled application code to a vendor SDK (dd-trace, New Relic agent). Switching vendors meant rewriting instrumentation. The OTel bet: one open API + OTLP wire format = swap backends without touching application code.
In practice, OTel APIs cover traces and metrics well. The OTel Logs API stabilised in late 2023; SDKs in late 2024. Semantic Conventions for HTTP and database are GA; messaging and FaaS are still maturing. The honest senior take: OTel is the right bet for new instrumentation in 2026 if portability matters. OTel SDKs coexist with vendor SDKs during transition.
- Honeycomb event ingestion ceiling per dataset
- ~1B events / hour
- ClickHouse-based observability typical compression
- 10:1 to 30:1
- Wide-event row size, fully-instrumented
- ~2–10 KB / event
- OTLP-gRPC vs HTTP JSON wire size
- ~50–70% smaller, ~2–5x faster encode
- OTel Collector throughput (commodity box)
- ~50–200k spans/sec
- Dual-write migration window (typical)
- 60–90 days
A 40-service team spends ~2 engineer-weeks/quarter on cardinality budgets and ~3 hours/incident on cross-pillar pivots. Which architecture is the better fit?
What is the primary reason long-retention low-cardinality metrics are sometimes cheaper to keep in a 1.0 TSDB than in a 2.0 wide-event store?
- 01Articulate the 1.0 vs 2.0 cost difference in billing terms, and name the workload where each is the better economic choice.
- 02What are the three common failure modes during a 1.0 to 2.0 migration?
- 03What is the irreversible engineering cost of staying on 1.0 when cardinality budgets are routinely exceeded?
The three-pillar taxonomy emerged from late-2000s storage cost cliffs: you could not afford raw event retention (metrics), real-time aggregation (logs), or full-fidelity causality on every request (traces). With columnar storage and cheap object storage, wide structured events — one record per unit of work with all attributes — collapse all three into one queryable surface. 2.0 pricing is ~traffic-month plus dimensionality; cardinality is no longer a cost driver, eliminating the productivity tax of label budgets. 2.0 wins for incident response and ad-hoc investigation at 30+ services. 1.0 wins for very-long-retention low-cardinality dashboards where pre-aggregation is still the cheapest cold-storage shape. Many mature shops run both, and the migration path is dual-write over 60–90 days.
appears again in167
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What an index is and how it speeds up queriesjunior
- The leading-column rule and composite index designmiddle
- Partial, expression, and covering indexesmiddle
- Index types: GIN, GiST, BRIN, Hash, Bloom, and HOT updatesmiddle
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- Index design exercise: full-text search strategysenior
- EXPLAIN and execution plans: what the planner decides and whyjunior
- Scan types: Seq, Index, Bitmap, Index-Onlymiddle
- Join algorithms and the row-estimate cascademiddle
- pg_statistic, ANALYZE, and production observabilitymiddle
- Extended statistics: fixing correlated-column estimate failuressenior
- Plan cache, cost-constant tuning, and planner internalssenior
- Production failure modes and plan stabilitysenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Migration failure taxonomy and production disciplinesenior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- Why profile first: measure where time actually goesjunior
- Amdahl''''s law and self-time: the ceiling on every speedup you can shipmiddle
- The measurement loop: microbench, macrobench, prod profile, observer effectmiddle
- Reading flame graphs: shapes, per-language profilers, and the 60-second scanmiddle
- Statistical baselines: why one run is not a measurementmiddle
- Profiler history and microbenchmark pitfalls: Knuth to GWPsenior
- Hardware counters, cold-start profiles, and profile securitysenior
- Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patternssenior
- What makes a hot path: symptom vs causejunior
- Five shapes of hotspot: CPU, alloc, cache, lock, syscallmiddle
- Reading parent and child chains: where to apply the fixmiddle
- JIT deopt, the fix-and-verify loop, and PR-time profilingmiddle
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Memory hierarchy: why the same O(N) loop can be 17x slowerjunior
- Row-major vs column-major: access order and the 9x gapjunior
- Branch prediction and branchless codemiddle
- Hardware prefetcher, TLB, and memory-level parallelismsenior
- GC basics: what the runtime taxes you forjunior
- GC algorithms: generational, concurrent, and per-runtimemiddle
- GC tradeoffs: pause, throughput, heap — and object poolingmiddle
- GC tuning: pacing, heap shape, and allocation observabilitymiddle
- GC internals: tri-color invariant, write barriers, and per-runtime deep-divessenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- N+1: one logical operation, many round-tripsjunior
- Fix families: JOIN, IN, preload, and DataLoadermiddle
- Detecting N+1: query logs, APM traces, and CI gatesmiddle
- DataLoader: batching across resolver treesmiddle
- Cross-protocol N+1: HTTP fan-out and Redis MGETmiddle
- N+1 at scale: pool exhaustion, plan changes, and denormalisationsenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- What a bundle actually costs: download, parse, compile, executejunior
- Core Web Vitals: LCP, INP, and CLSmiddle
- Code splitting: route-level, component-level, vendor splittingmiddle
- Tree shaking and compression: removing what you don''''t usemiddle
- Third-party scripts: the silent budget killermiddle
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior