The OTel Collector: receivers, processors, exporters, and deployment patterns

A new backend requirement lands: route traces to vendor-A, logs to vendor-B, and redact PII from both, starting next Monday. Without a Collector, that is three separate application deploys across 50 teams. With a Collector, it is one YAML change and one Collector restart.

Receivers, processors, exporters

The Collector is a YAML-configured pipeline with three stages:

Receivers accept incoming telemetry:

• otlp — gRPC (port 4317) and HTTP (port 4318), the primary receiver for OTel-instrumented services
• filelog — tails log files from the filesystem (useful for legacy apps that write to stdout/files)
• prometheus — scrapes /metrics endpoints (bridges Prometheus exporters into OTel)
• kafka — reads telemetry from Kafka topics
• Vendor-specific receivers for non-OTel data

Processors transform records in-flight:

• memory_limiter — drops new records when the Collector is above a RAM threshold; prevents OOM by design
• batch — groups records into efficient batches (fewer network round trips to exporters)
• resource — adds service.name and other Resource attributes (e.g., inject deployment.environment=production)
• attributes — redacts, renames, adds fields; used for PII scrubbing and Semantic Convention enforcement
• tail_sampling — keeps/drops traces based on the full trace context (errors, latency, business criteria) — covered in the next lesson
• transform — general-purpose transformations via OTTL (OpenTelemetry Transformation Language)
• k8sattributes — enriches spans/logs with Kubernetes pod, namespace, node, and deployment metadata

Exporters ship records to backends:

• otlp — to another OTel-aware backend (another Collector, Grafana Tempo, Jaeger, etc.)
• datadog — vendor-specific
• prometheusremotewrite — to Prometheus or Grafana Mimir
• loki — to Grafana Loki for logs
• Vendor-specific exporters for New Relic, Honeycomb, Splunk, Elastic, etc.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 10s
    send_batch_size: 512
  attributes:
    actions:
      - key: user.email
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, attributes]
      exporters: [otlp/tempo]

Three deployment patterns

Agent — a Collector runs as a sidecar (one per pod) or DaemonSet (one per node). Receives telemetry from the local application, does minimal processing (resource enrichment, batching), exports directly to the backend. Cheapest in network hops, hardest to run tail sampling (each agent sees only one node’s traffic).

Gateway — applications export directly to a central pool of Collector replicas that does heavy processing (tail sampling, redaction, multi-backend routing). Easier to scale processing centrally, but every span crosses the network from the app to the gateway.

Agent-to-gateway (the production-canonical pattern) — a DaemonSet agent on every node does minimal local processing and forwards via OTLP to a centralised gateway pool that handles tail sampling, redaction, and routing. The agent adds Kubernetes metadata (pod, namespace, node) via the k8sattributes processor before forwarding.

In the agent-to-gateway pattern, the agent’s loadbalancing exporter routes by trace_id hash to ensure all spans of a trace land on the same gateway replica — necessary for tail sampling, covered in the next lesson.