Observability
Vendor neutrality, eBPF instrumentation, the Operator, and browser/serverless OTel
A team uses a vendor-distro Collector with custom proprietary processors. They believe they are OTel-neutral because their application emits OTLP. A year later, migrating to a new vendor requires rewriting not just the Collector config but the custom processing logic that only exists in the old vendor’s distribution. Vendor-neutrality was silently eroded.
The vendor-neutrality trap
Many vendors ship “OTel-compatible” Collector distributions — Datadog Agent, Splunk OTel Collector, New Relic distro — that include proprietary processors and exporters. Using one is fine, but with two caveats that frequently create silent lock-in.
(1) Proprietary processors: if you rely on a processor that exists only in vendor-X’s distribution (e.g., a vendor-specific routing-by-cost feature, a vendor-specific RUM correlator), your Collector config is no longer portable. Moving to vendor-Y means rewriting that processor — it does not exist upstream.
(2) Edge OTLP: the contract that defines vendor-neutrality is “does the application edge emit OTLP?” If application code calls the vendor SDK directly (dd-trace, New Relic agent) and the data is in vendor format at the application boundary, you are not vendor-neutral even if a Collector is in the middle — because the migration cost lives in the application code, in every service.
The Honeycomb position: vendor distros are fine for collection-tier conveniences, but if application instrumentation is vendor-specific, you face an instrumentation-cost migration next time.
The portability audit: run quarterly. Grep application code for vendor-SDK imports. Inspect Collector YAML for processor names that exist only in a vendor distro. Review Semantic Convention adherence per service. Vendor-neutrality is not binary — it is a posture maintained by discipline.
OTel and the wide-event question
OTel’s stable data model is still three-pillar — traces, metrics, logs as separate APIs and OTLP message types. But the industry is converging on a more unified model. A wide-event store (Honeycomb, ClickHouse) can ingest OTel logs (with trace context attached) and treat them as one stream. Future OTel work — the Profiles signal, wide-event correlation discussions in the spec working group — moves toward a more unified data model.
For senior teams in 2026, the implication: adopting OTel now is forward-compatible with both 1.0 (pillar-separated) backends and 2.0 (wide-event) backends. Switching backends later is a Collector config change, not an instrumentation rewrite.
eBPF auto-instrumentation: the no-code path
A growing class of OTel-compatible instrumentation runs in the kernel via eBPF — Grafana Beyla, Pixie, Datadog USM. These tools attach kernel probes that observe syscalls, HTTP traffic on sockets, and gRPC RPCs without any application code or library changes.
What eBPF sees: HTTP + gRPC + DB socket calls, latency, status codes. Service identity from the cgroup (service.name inferred from the process name or cgroup label). OTel-shaped spans with Semantic Conventions attribution.
What eBPF cannot see: business attributes — customer.segment, feature_flag, batch_size — because these live inside the application, not in the syscall. Slow-tail attribution to specific application logic is harder.
Pattern in practice: eBPF for the breadth (free coverage of every service, including vendor binaries you cannot modify), SDK instrumentation for the depth (business spans and attributes). Both emit OTLP to the same Collector, both speak Semantic Conventions, both show up in the same dashboards.
| Approach | Coverage | Business attributes? | Code changes? |
|---|---|---|---|
| eBPF (Beyla, Pixie) | HTTP + gRPC + DB sockets | No | None |
| OTel SDK auto-instrumentation | Framework calls (HTTP, DB, queues) | No (framework-level only) | Agent flag or require hook |
| Manual SDK instrumentation | Business operations | Yes | Yes — per business operation |
The OTel Operator: instrumentation as platform responsibility
The OTel Operator (a Kubernetes Operator under the CNCF umbrella) manages the Collector deployment declaratively.
OpenTelemetryCollector CRD: defines agent and gateway Collector specs in YAML. The Operator creates the DaemonSet, Deployment, and Services from the CRD; updating the CRD triggers a rolling restart with zero downtime.
Instrumentation CRD: configures auto-instrumentation injection via annotations on pods. Example:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators: [tracecontext, baggage]
sampler:
type: parentbased_traceidratio
argument: "0.25"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latestA pod annotated with instrumentation.opentelemetry.io/inject-java: "true" gets the Java Agent injected via an initContainer at startup — no Dockerfile changes, no code changes.
Strategic value: instrumentation becomes a platform-team responsibility, not a per-service responsibility. A new service deployed with the right annotation gets full auto-instrumentation without touching its Dockerfile or code. Mature platforms make this the default; opting out requires an explicit annotation.
Browser and serverless: OTel/HTTP and cold-start constraints
Browser: The OTel JavaScript SDK supports the browser environment with @opentelemetry/sdk-trace-web — instrumenting fetch, XMLHttpRequest, document load, user interactions. Browsers emit OTLP/HTTP (not gRPC; gRPC is browser-incompatible) to a Collector endpoint configured with CORS. The resulting browser spans join the server-side trace via the W3C traceparent header propagated on outgoing requests — true end-to-end traces from user click to database query.
Lambda: Cold-start cost of OTel SDK initialization is significant (50-200 ms for full Lambda instrumentation). The standard approach uses the AWS OTel Lambda Layer — the layer initializes the SDK in the runtime extension and amortises startup cost across invocations.
Cloudflare Workers / Vercel Edge: Still maturing OTel support, with limitations on async-context propagation in the V8 isolate model.
Why this works
Why does gRPC not work in browsers? gRPC requires HTTP/2 framing at the application layer — specifically, it relies on the ability to control HTTP/2 stream framing directly. Browsers do not expose raw HTTP/2 framing to JavaScript; the Fetch API only supports HTTP/1.1 and HTTP/2 at the network level but not the gRPC-specific framing. gRPC-Web (a proxy-based protocol) is one workaround, but OTel chose OTLP/HTTP over HTTP/1.1 as the browser transport for maximum compatibility — it works through any CORS-enabled Collector endpoint without requiring a proxy.
A team uses a vendor-distro Collector with a custom vendor-specific routing processor. They consider themselves OTel-neutral because application code emits OTLP. Senior review — agree or disagree?
What does eBPF instrumentation (Grafana Beyla, Pixie) observe that the OTel Java Agent cannot, and what does the Java Agent observe that eBPF cannot?
- 01Articulate precisely what makes OTel 'vendor-neutral' and the two most common ways this neutrality silently erodes in production.
- 02What is the strategic value of the OTel Operator, and what does it unlock for a platform team?
- 03Why does browser OTel use OTLP/HTTP instead of OTLP/gRPC, and what does this mean for end-to-end trace stitching?
True vendor-neutrality requires OTLP at the application edge — not just “we use a Collector.” Two silent erosion paths: (1) vendor SDK in application code (dd-trace, New Relic agent) means the migration cost lives in every service’s code, not just the Collector config; (2) proprietary Collector processors lock the policy plane to one vendor’s distribution. Run a quarterly portability audit to detect both. eBPF instrumentation (Grafana Beyla, Pixie) attaches kernel probes to observe HTTP, gRPC, and DB sockets for any process without code changes — business attributes invisible to eBPF are added by SDK manual instrumentation. The OTel Operator manages both the Collector deployment and per-service instrumentation injection via CRDs — a pod annotation triggers auto-instrumentation with no Dockerfile or code changes. Browser OTel uses OTLP/HTTP (gRPC is browser-incompatible) and propagates W3C traceparent for true end-to-end traces. Lambda uses a Layer to amortise the 50-200 ms OTel cold-start cost.
appears again in202
- Federation and lookahead: batching beyond DataLoadermiddle
- Senior GraphQL API: scheduling contract, tenant isolation, observabilitysenior
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Invalidation, dirty bits, and containmiddle
- Compositor layers: promotion, overlap, and GPU memorymiddle
- Production observability: LoAF, INP, and the full attack surfacesenior
- Hidden classes, transition trees, and memory layoutmiddle
- V8 in production: isolates, pointer compression, and real failuressenior
- What workers are and why they existjunior
- Web worker mechanics: dedicated, shared, and OffscreenCanvasmiddle
- Structured clone and transferablesmiddle
- SharedArrayBuffer, Atomics, and cross-origin isolationsenior
- Worker pools, Comlink, and production observabilitysenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- Lock and single-flight: bounding concurrent rebuildsmiddle
- Stale-while-revalidate and CDN request coalescingmiddle
- Detecting stampedes and designing TTL for productionmiddle
- Metastable failure, fencing tokens, and production postmortemssenior
- What a relation is: tables, rows, keys, and constraintsjunior
- Constraints, keys, and Postgres data typesmiddle
- JSONB, arrays, and when a side table winsmiddle
- Schema integrity: deferral, versioning, and production failure modessenior
- What an index is and how it speeds up queriesjunior
- The leading-column rule and composite index designmiddle
- Partial, expression, and covering indexesmiddle
- Index types: GIN, GiST, BRIN, Hash, Bloom, and HOT updatesmiddle
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- Index design exercise: full-text search strategysenior
- EXPLAIN and execution plans: what the planner decides and whyjunior
- Scan types: Seq, Index, Bitmap, Index-Onlymiddle
- Join algorithms and the row-estimate cascademiddle
- pg_statistic, ANALYZE, and production observabilitymiddle
- Extended statistics: fixing correlated-column estimate failuressenior
- Plan cache, cost-constant tuning, and planner internalssenior
- Production failure modes and plan stabilitysenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Migration failure taxonomy and production disciplinesenior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Where data fetching happens — and why it decides LCPjunior
- React Server Components and Suspense streamingmiddle
- Senior internals: RSC payload, caching layers, and production failure modessenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- The IP envelopejunior
- Reading the IP headermiddle
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- What TLS does and why it existsjunior
- Key schedule, SNI, ALPN, and extensionssenior
- 0-RTT defenses, ECH, hybrid PQ, and production TLSsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- Why profile first: measure where time actually goesjunior
- Amdahl''''s law and self-time: the ceiling on every speedup you can shipmiddle
- The measurement loop: microbench, macrobench, prod profile, observer effectmiddle
- Reading flame graphs: shapes, per-language profilers, and the 60-second scanmiddle
- Statistical baselines: why one run is not a measurementmiddle
- Profiler history and microbenchmark pitfalls: Knuth to GWPsenior
- Hardware counters, cold-start profiles, and profile securitysenior
- Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patternssenior
- What makes a hot path: symptom vs causejunior
- Five shapes of hotspot: CPU, alloc, cache, lock, syscallmiddle
- Reading parent and child chains: where to apply the fixmiddle
- JIT deopt, the fix-and-verify loop, and PR-time profilingmiddle
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Memory hierarchy: why the same O(N) loop can be 17x slowerjunior
- Row-major vs column-major: access order and the 9x gapjunior
- Branch prediction and branchless codemiddle
- Hardware prefetcher, TLB, and memory-level parallelismsenior
- GC basics: what the runtime taxes you forjunior
- GC algorithms: generational, concurrent, and per-runtimemiddle
- GC tradeoffs: pause, throughput, heap — and object poolingmiddle
- GC tuning: pacing, heap shape, and allocation observabilitymiddle
- GC internals: tri-color invariant, write barriers, and per-runtime deep-divessenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- N+1: one logical operation, many round-tripsjunior
- Fix families: JOIN, IN, preload, and DataLoadermiddle
- Detecting N+1: query logs, APM traces, and CI gatesmiddle
- DataLoader: batching across resolver treesmiddle
- Cross-protocol N+1: HTTP fan-out and Redis MGETmiddle
- N+1 at scale: pool exhaustion, plan changes, and denormalisationsenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- What a bundle actually costs: download, parse, compile, executejunior
- Core Web Vitals: LCP, INP, and CLSmiddle
- Code splitting: route-level, component-level, vendor splittingmiddle
- Tree shaking and compression: removing what you don''''t usemiddle
- Third-party scripts: the silent budget killermiddle
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior
- At-most-once, at-least-once, exactly-once: the three delivery contractsjunior
- Consumer-side dedup: the cheapest path to exactly-once processingmiddle
- Exactly-once in production: impossibility proof, hybrid patterns, and real incidentssenior
- What OAuth is and why passwords are not the answerjunior
- Authorization code flow with PKCEmiddle
- Sender-constrained tokens: DPoP and mTLSsenior
- OAuth in production: audience attacks, observability, and real failuressenior