Operating the OTel Collector: reliability, version skew, failure modes, and governance

The Collector fails. Application errors spike but no alerts fire, no traces appear in the backend. The on-call engineer checks dashboards — all green, because the dashboards depend on the same Collector that just failed. OTel self-monitoring is not optional.

Reliability patterns

HA gateway — minimum 3 replicas: A single gateway pod failure loses all in-flight spans buffered in that pod. Three replicas mean one-pod failure is survivable with client retries. Behind a Kubernetes Service or cloud load balancer; the loadbalancing exporter on agents uses the service endpoint so scale-up/down is transparent to agents.

Persistent queue — the file_storage extension provides a disk-backed buffer that survives Collector restarts. Configure it on the gateway’s export pipelines to absorb 5-15 minutes of backend slowdown without dropping spans:

extensions:
  file_storage:
    directory: /var/otel/queue

exporters:
  otlp/primary:
    endpoint: backend:4317
    sending_queue:
      storage: file_storage
      queue_size: 10000

Health checks — liveness and readiness probes against the health_check extension on port 13133. Do not let a slow or overloaded Collector be considered ready; it will continue receiving spans it cannot process.

Self-monitoring — scrape the Collector’s /metrics endpoint (port 8888) and alert on:

• otelcol_processor_dropped_spans rate > 0 — memory_limiter engaging; warn immediately
• otelcol_receiver_refused_spans rate > 0 — back-pressure at the receiver; correlates with memory_limiter
• otelcol_exporter_send_failed_spans rate > 0 — backend connectivity problem
• otelcol_exporter_queue_size / queue capacity > 80% — exporter backlog building; backend slow
• otelcol_processor_tail_sampling_count_traces_on_memory vs num_traces — buffer exhaustion approaching
• process_resident_memory_bytes vs configured limit — approaching OOM

Resource sizing — a commodity gateway pod (4 CPU, 8 GB RAM) handles ~100-200k spans/sec with tail sampling. Size for peak + 2× headroom. Set CPU requests low and RAM requests/limits tight (memory_limiter should engage before Linux OOM killer).

Version skew and stability strategy

OTel is many independently versioned components: the spec (v1.x), each language SDK (varies), each Collector binary (v0.x with rapid releases), each Semantic Convention domain (HTTP 1.x, DB 1.x, etc.).

Compatibility: SDKs are forward-compatible with newer Collectors across multiple minor versions; OTLP is stable. The Collector has a notion of stable and beta components — production setups stick to stable receivers, processors, exporters.

Strategy:

1. Pin SDK and Collector versions in deployment manifests
2. Upgrade quarterly with a canary before fleet-wide rollout
3. Track Semantic Convention versions per service so dashboards know what attribute names to expect
4. Use the OTel Operator for Collector upgrades: CRD update triggers a rolling restart, zero downtime

Production failure modes

(a) Collector OOM under tail sampling: Gateway buffer grows past memory limit because decision_wait is too long or trace volume spiked. Mitigation: memory_limiter before tail_sampling; alert on dropped_spans; right-size num_traces for peak rate × decision_wait × 2.

(b) Tail-sample re-routing on scale events: Gateway pool scales up, loadbalancing exporter’s hash ring re-shuffles, in-flight traces lose some spans. Mitigation: pre-warm new pods, scale conservatively, use longer convergence windows on the loadbalancing exporter.

(c) OTLP version mismatch: A Collector upgraded ahead of SDKs encounters an unknown field in a newer OTLP proto; may silently drop attributes or the whole record. Mitigation: SDK and Collector compatibility matrix; staged upgrades; never upgrade Collector ahead of the SDKs it receives from.

(d) Auto-instrumentation footprint regression: A new minor version of the OTel Java Agent adds an instrumentation that slows a critical library. Mitigation: canary the agent upgrade; monitor p99 latency on the affected service; use per-instrumentation opt-out flags (OTEL_INSTRUMENTATION_X_ENABLED=false).

(e) Cardinality leak via auto-instrumentation: Auto-instrumented HTTP client adds url.full (the raw URL with query parameters) as an attribute, exploding cardinality at the metrics backend. Mitigation: configure the instrumentation to use http.route (templated) instead of url.full; strip query strings via an attributes processor at the Collector.

Semantic Convention governance

Semantic Conventions are how every team’s telemetry composes at fleet scale. Governance failures are expensive:

• Team-A names a field route
• Team-B names it http_route
• Team-C names it http.route (the correct Semantic Convention name)
• Cross-team dashboards use http.route — teams A and B are invisible

Pattern: platform team publishes a per-language wrapper that pre-configures Semantic Convention attribute extraction. New services import the wrapper; CI lint rejects raw SDK usage in new code. The wrapper handles:

• HTTP route extraction (matched template, not raw URL)
• DB system tagging (db.system=postgresql, not “psql”)
• Redaction deny-lists
• Trace-context mixins for logs

Quarterly audit: check top-10 most-used attribute names per service for Semantic Convention drift. The audit output is the platform team’s backlog.