Operating the OTel Collector: reliability, version skew, failure modes, and governance

Reliability patterns

When the Collector fails, you lose visibility at the exact moment you need it most — this lesson tells you which five practices prevent that and which metrics tell you when you are about to hit the wall before you actually do.

HA gateway — minimum 3 replicas: A single gateway pod failure loses all in-flight spans buffered in that pod. Three replicas mean one-pod failure is survivable with client retries. Behind a Kubernetes Service or cloud load balancer; the loadbalancing exporter on agents uses the service endpoint so scale-up/down is transparent to agents.

Persistent queue — the file_storage extension provides a disk-backed buffer that survives Collector restarts. Configure it on the gateway’s export pipelines to absorb 5-15 minutes of backend slowdown without dropping spans:

extensions:
  file_storage:
    directory: /var/otel/queue

exporters:
  otlp/primary:
    endpoint: backend:4317
    sending_queue:
      storage: file_storage
      queue_size: 10000

Health checks — liveness and readiness probes against the health_check extension on port 13133. Do not let a slow or overloaded Collector be considered ready; it will continue receiving spans it cannot process.

Self-monitoring — scrape the Collector’s /metrics endpoint (port 8888) and alert on:

• otelcol_processor_dropped_spans rate > 0 — memory_limiter engaging; warn immediately
• otelcol_receiver_refused_spans rate > 0 — back-pressure at the receiver; correlates with memory_limiter
• otelcol_exporter_send_failed_spans rate > 0 — backend connectivity problem
• otelcol_exporter_queue_size / queue capacity > 80% — exporter backlog building; backend slow
• otelcol_processor_tail_sampling_count_traces_on_memory vs num_traces — buffer exhaustion approaching
• process_resident_memory_bytes vs configured limit — approaching OOM

Resource sizing — a commodity gateway pod (4 CPU, 8 GB RAM) handles ~100-200k spans/sec with tail sampling. Size for peak + 2× headroom. Set CPU requests low and RAM requests/limits tight (memory_limiter should engage before Linux OOM killer).

Version skew and stability strategy

OTel is many independently versioned components: the spec (v1.x), each language SDK (varies), each Collector binary (v0.x with rapid releases), each Semantic Convention domain (HTTP 1.x, DB 1.x, etc.).

Compatibility: SDKs are forward-compatible with newer Collectors across multiple minor versions; OTLP is stable. The Collector has a notion of stable and beta components — production setups stick to stable receivers, processors, exporters.

Strategy:

1. Pin SDK and Collector versions in deployment manifests
2. Upgrade quarterly with a canary before fleet-wide rollout
3. Track Semantic Convention versions per service so dashboards know what attribute names to expect
4. Use the OTel Operator for Collector upgrades: CRD update triggers a rolling restart, zero downtime

Production failure modes

(a) Collector OOM under tail sampling: Gateway buffer grows past memory limit because decision_wait is too long or trace volume spiked. Mitigation: memory_limiter before tail_sampling; alert on dropped_spans; right-size num_traces for peak rate × decision_wait × 2.

(b) Tail-sample re-routing on scale events: Gateway pool scales up, loadbalancing exporter’s hash ring re-shuffles, in-flight traces lose some spans. Mitigation: pre-warm new pods, scale conservatively, use longer convergence windows on the loadbalancing exporter.

(c) OTLP version mismatch: A Collector upgraded ahead of SDKs encounters an unknown field in a newer OTLP proto; may silently drop attributes or the whole record. Mitigation: SDK and Collector compatibility matrix; staged upgrades; never upgrade Collector ahead of the SDKs it receives from.

(d) Auto-instrumentation footprint regression: A new minor version of the OTel Java Agent adds an instrumentation that slows a critical library. Mitigation: canary the agent upgrade; monitor p99 latency on the affected service; use per-instrumentation opt-out flags (OTEL_INSTRUMENTATION_X_ENABLED=false).

(e) Cardinality leak via auto-instrumentation: Auto-instrumented HTTP client adds url.full (the raw URL with query parameters) as an attribute, exploding cardinality at the metrics backend. Mitigation: configure the instrumentation to use http.route (templated) instead of url.full; strip query strings via an attributes processor at the Collector.

These five failure modes share a pattern: each is detectable before it causes an outage — through self-monitoring metrics, canary upgrades, or cardinality alerts — but only if you instrument the Collector itself as rigorously as you instrument your application.

Semantic Convention governance

Semantic Conventions are how every team’s telemetry composes at fleet scale. Governance failures are expensive:

• Team-A names a field route
• Team-B names it http_route
• Team-C names it http.route (the correct Semantic Convention name)
• Cross-team dashboards use http.route — teams A and B are invisible

Pattern: platform team publishes a per-language wrapper that pre-configures Semantic Convention attribute extraction. New services import the wrapper; CI lint rejects raw SDK usage in new code. The wrapper handles:

• HTTP route extraction (matched template, not raw URL)
• DB system tagging (db.system=postgresql, not “psql”)
• Redaction deny-lists
• Trace-context mixins for logs

Quarterly audit: check top-10 most-used attribute names per service for Semantic Convention drift. The audit output is the platform team’s backlog.