Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patterns

An SLO burn alert fires. In 2020, the on-call opens a terminal, runs a manual profile capture, waits 60 seconds, and stares at a flame graph that may or may not represent the traffic causing the burn. In 2026, they click the alert, click the slow trace, click the span — and the flame graph filtered to that exact trace-id is already there.

Continuous profiling: costs and tradeoffs

Always-on profiling at 2-5% overhead is the modern default. The cost is non-zero and compounds across a fleet.

CPU overhead: 200-service fleet at 100 Hz, 2% overhead = the equivalent of running ~4 extra services worth of CPU permanently.

Storage overhead: 30-second profiles at ~200 KB compressed × 240 per hour × 24 hours × 200 services ≈ 230 GB/day raw. Pyroscope 2.0’s symbol deduplication cuts this 3x. Tiered retention (7 days fine, 30 days downsampled, 90 days summary) cuts another 5x in aggregate. Net cost on object storage: ~$25-100/month for 200 services.

The rollout pattern: pay the cost only when the team has the discipline to use the data. A small team starting out may get more value from on-demand profiling (perf record, pprof endpoint) and adopt continuous profiling later. The graduation criterion: reaching-for-the-profile is already a reflex. If the team has to remember to look, continuous profiling becomes decoration.

Profile-trace correlation

The integrated triage workflow:

SLO burn alert → click → traces filtered to the burn window → pick a slow trace → click span → profile filtered to that span’s trace-id → flame graph for the specific request.

Sub-30-second drill from page-out to git blame.

The bridge: trace-id stamping inside profile samples. Each stack sample carries the active trace-id at the moment of sampling. At query time the backend joins profile samples to the trace. OpenTelemetry’s profile signal (beta as of 2026) standardises this stamping.

Production implementations:

• Grafana Tempo + Pyroscope
• Datadog APM + Continuous Profiler
• Honeycomb + OTel profile signal

Without trace-id stamping, the on-call must guess which profile sample corresponds to the slow request. With it, the answer is one filter away.

Profile-driven CI gates

Continuous profiling enables pre-merge regression detection.

Pattern: every PR triggers a canary deploy. A 5-minute load test under representative traffic captures a CPU profile and an allocation profile. The CI job diffs against main’s baseline. If any function’s CPU share grows by more than a configured threshold (typically 10% absolute or 30% relative), the PR is flagged for review.

Implementations: Pyroscope’s compare API, Datadog’s deploy comparison, custom pipelines using Go’s pprof diff or async-profiler’s JFR diff.

Payoff: regressions that would have hit production at p99 are caught in CI. Mean time to detect on perf regressions drops from days (production monitoring eventually flags) to minutes (CI fails the PR).

Operational challenges: (1) Noisy baselines — main itself is changing; refresh the baseline weekly, not nightly. (2) Variance thresholds — start loose (≥30% absolute) and tighten over months. (3) Synthetic load mismatch — extend the load profile based on missed regressions caught in production. (4) Engineer pushback — provide a /perf-override escape hatch requiring manager sign-off and logged for audit.

Production failures: profile-first paid off

Stripe (2021): service at 80% CPU. Team guessed retries. Profile showed 60% of CPU in a JSON parser called on every health-check 100 times/second. Fix: cache the parsed config. CPU dropped to 30% in 8 minutes.

GitHub (2020): Ruby workers OOMing. Allocation profile pointed at template-rendering allocating 200 MB per request.

Discord (2020): chat tail latency. JSON serialisation switch dropped p99 tail.

Counter-example (2019 startup): two months rewriting a Postgres query for a “slow” admin page. Profile finally showed 90% in a third-party widget. SQL change moved nothing.

Pattern: with the profile, the bottleneck is named in minutes. Without it, teams burn weeks on the wrong code.

Anti-patterns

Five common anti-patterns in performance work:

1. Optimising the cold path — a function called once at startup gets a 100x rewrite while the per-request hot path is ignored. Catch: “calls per second × per-call cost”, not just per-call cost.
2. Microbench-driven optimisation — function X is 10x faster in isolation; production is 1.03x. Catch: production profile share before any microbench rewrite.
3. “We cannot profile production, it is too expensive” — an excuse to skip the only honest measurement. Catch: continuous profiling overhead is 2-5%, well below the gain from finding any real bottleneck.
4. Single-run reporting — “this PR is 12% faster” with one measurement. Catch: insist on distributions.
5. Regression-by-feature-flag — a perf improvement is gated behind a flag and never enabled. Catch: the CI profile gate must test flag-enabled runs; otherwise the improvement is permanent dead code.

# Production metrics (5-minute window)
checkout_p99_ms          580
checkout_p99_ms_prev     820  # before deploy
cpu_pct                  62
cpu_pct_prev             58

# go tool pprof -diff_base baseline.cpu prod.cpu
File: checkout
Type: cpu
Showing nodes accounting for -3.20s, 1.15% of -278.5s total

    flat  flat%   sum%        cum   cum%
  -1.80s  0.64%  0.64%    -1.80s  0.64%  net/http.(*conn).serve
  -1.40s  0.50%  1.15%    -1.40s  0.50%  encoding/json.Marshal
  +0.05s 0.018%  1.13%    +0.05s 0.018%  myapp/handlers.Checkout
  ... (other shifts < 0.5s)