Sampling strategies and log cost

A team turns on structured logging. Six weeks later their log bill is three times higher than their compute bill. Every service, every request, every INFO line — indexed and billed. Logging costs more than serving traffic. The service is fine. The sampling policy is missing.

The cost equation

The cost of a log line is paid in three places: at write time (CPU + RAM for serialization), in transit (network egress + collector capacity), and at the backend (ingest GB + indexed-event count + retention bytes).

A modest service emitting one log line per request handles 1 MB/s at 1000 req/s — about 86 GB/day. At hosted log pricing ($0.10/GB ingest plus indexed-event cost), the bill compounds fast across dozens of services.

Three sampling strategies

Success-path sampling keeps 1-in-N INFO lines for high-volume successful events while keeping 100% of WARN and ERROR. Typical N is 10 to 100. This is the first lever to pull: it cuts INFO volume by 90% without touching failure forensics.

Pattern-based sampling keeps every distinct log pattern at full rate and samples the duplicates. Vector and Fluent Bit ship sampling filters that hash the message template — so a “retry attempt 1 of 5” pattern is kept at 1-in-100 while “payment_declined” (rare) stays at 100%.

Tail sampling for logs mirrors the trace pattern: buffer logs for a request window, then decide based on outcome — keep all logs for failed requests, sample the successful ones. This is the most powerful strategy but requires a stateful buffer at the collector tier. It guarantees zero loss of failure context while discarding up to 99% of success-path volume.

The shipping pipeline

Logs travel through three stages: emit (the application writes JSON to stdout or a logger SDK), collect (a sidecar agent or DaemonSet reads stdout, parses JSON, batches, applies sampling), ship (OTLP-HTTP or native protocol to the backend).

The collector layer — Fluent Bit, Vector, OTel Collector with the filelog receiver — does three things you do not want in the application: backpressure (buffer on disk if the backend is slow), enrichment (attach resource attributes from pod metadata), and redaction (strip PII patterns before they leave the host).

Production rule: emit JSON to stdout, let the platform handle everything after that.

Retention tiering: hot, warm, cold

Indexed log storage at $0.10-$1.00/GB-month is too expensive for multi-month retention at scale. Mature stacks tier:

• Hot (last 7-15 days, fully indexed, sub-second query — Datadog Standard, Loki recent)
• Warm (30-90 days, partially indexed or scan-only — Datadog Flex, Splunk Frozen-Searchable)
• Cold (compliance retention in S3 or equivalent at $0.023/GB-month, restorable but not directly queryable)

An incident under 7 days old runs against hot tier with full query power. An investigation into “what happened 6 months ago” needs warm-tier queries that may take minutes per scan and may not have every dimension indexed.