Error tracking, correlation and alert fatigue

Three pillars, one incident: correlate by id

Logs tell you what happened (a line per event), metrics tell you how much / how fast (the rates and percentiles from L04), traces tell you where in the call tree time and failures live (the spans from L03). Each is necessary; none is sufficient alone. The thing that turns three disconnected haystacks into one searchable incident is a shared join key stamped on all of them: a traceId propagated through the request, plus a requestId written onto every log line, attached to every error report, and carried on the trace.

// A correlation middleware/interceptor stamps the join key once, early.
// (Logging envelope is L01; trace propagation is L03 — here we only thread the IDS.)
@Injectable()
export class CorrelationInterceptor implements NestInterceptor {
  intercept(ctx: ExecutionContext, next: CallHandler) {
    const req = ctx.switchToHttp().getRequest();
    // reuse the inbound trace if a tracer set one, else mint a requestId
    const traceId = trace.getActiveSpan()?.spanContext().traceId;
    const requestId = req.headers['x-request-id'] ?? randomUUID();
    // make both available to the logger, the error reporter, and downstream calls
    return runWithContext({ traceId, requestId }, () => next.handle());
  }
}

With the key in place, an alert becomes a pivot, not an investigation: error in the tracker → read its traceId → open that exact trace → filter logs to that one requestId. You move from “something is wrong” to “this request, this span, these log lines” in three clicks. Without the join key you have a log search, a trace search, and an error list that share no vocabulary — three tools, three time-correlation guesses, and a much longer hour.

Error tracking: group by fingerprint, don’t drown

A log line per error drowns. The hundredth occurrence of the same NullPointerException is the hundredth identical line you scroll past. An error tracker (Sentry-style) inverts that: it computes a fingerprint from the error’s type and stack frames, then groups every occurrence with the same fingerprint into one issue, counts occurrences, and captures context — release, user, request, breadcrumbs — once per group. Ten thousand identical failures become one issue with a count of 10,000, not 10,000 lines.

In Nest, the global exception filter you built in L01 is the natural reporting point: it already catches every throw inside the request pipeline, so it’s where you hand the error to the tracker with the correlation context attached.

@Catch()
export class SentryExceptionFilter implements ExceptionFilter {
  catch(exception: unknown, host: ArgumentsHost) {
    const { traceId, requestId } = getContext();
    Sentry.withScope((scope) => {
      scope.setTag('traceId', traceId);       // <- links this issue to its trace
      scope.setTag('requestId', requestId);   // <- and to its exact log lines
      scope.setUser({ id: getUserId() });     // who hit it (id only, not PII)
      Sentry.captureException(exception);      // fingerprinted + grouped by Sentry
    });
    // ... still translate to the HTTP response as in L01
  }
}

Two attachments earn their keep. release — the deploy SHA, set once at boot (Sentry.init({ release })) — makes a spike map to a specific deploy: a new issue that appears the minute a3f9c1 shipped is a regression you can revert. traceId/requestId make every issue a one-click jump into the trace and the logs, which is the correlation from the previous section made concrete.

The unhandled-rejection trap

The global filter has a hard boundary: it only catches throws inside the request pipeline. An error thrown out-of-band — an unhandled promise rejection in a setInterval job, a throw inside an EventEmitter handler, a rejected promise nobody awaited — never enters Nest’s exception-handling path, so it never reaches the filter and never reaches the tracker. It vanishes. On modern Node an unhandledRejection can also terminate the process outright.

War-story: a nightly reconciliation job did void doReconcile() — fire-and-forget — and one night doReconcile rejected on a malformed row. The global filter never saw it (no request), Sentry never saw it (no filter), and on Node 18 the unhandled rejection took down the worker pod. The pod restarted, retried the same bad row, rejected again, crashed again — a silent crash-loop that ran for hours with zero error reports, because the one place errors were collected was a place this error could never reach.

// Out-of-band errors need PROCESS-level handlers — the filter cannot see them.
process.on('unhandledRejection', (reason) => {
  Sentry.captureException(reason);           // report it instead of losing it
});
process.on('uncaughtException', (err) => {
  Sentry.captureException(err);
  Sentry.flush(2000).finally(() => process.exit(1)); // report, drain, then exit
});

The rule: the global filter covers the request pipeline; process handlers cover everything else. You need both, or whole classes of failure are invisible.

Alert fatigue: alert on the signal, not the noise

The failure mode is alerting on every error, or on a raw error count. Both generate noise: a transient blip, a handled 404, a retried-then-succeeded timeout each page someone. When everything pages, people mute, and the mute is permanent. The discipline has four moves:

• Alert on rate / SLO burn, not count (L04). A burn-rate alert fires when you’re consuming the error budget faster than the SLO tolerates — it ignores a steady safe trickle and catches a real cliff.
• Separate “log it” from “page someone.” Handled, expected conditions (a validation 400, a known retryable timeout) get logged and counted, never paged. Only unexpected, user-impacting failures page.
• Set severities. Page on SEV1/SEV2; route everything else to a dashboard or a digest, not a human’s phone at 3am.
• Use the tracker’s grouping. One new fingerprint is one notification, not 10,000. “A new issue appeared” is signal; “an existing issue ticked up by one” usually isn’t.

Together these four moves turn an over-loud channel into a trustworthy one: drop the safe trickle, dedupe the repeated crashes, and without the severity discipline the first three still generate noise that gets muted.

PII and secrets: scrub before egress

Before you wire up Sentry or any error tracker, ask yourself: what’s in a typical request body or Authorization header in your app? If the answer includes tokens, emails, or any user data — and it almost always does — then shipping the raw request to a third party is a breach you logged yourself. Error context is a leak waiting to happen. Breadcrumbs and request snapshots can capture request bodies, headers (including Authorization!), emails, and tokens — and an error tracker is a third party. Shipping a raw request to it can mean shipping a live bearer token or a customer’s PII to your error vendor. That’s a breach, logged by you, on purpose. You must scrub before anything leaves the process: a beforeSend hook that redacts known-sensitive fields, a deny-list for headers, and token/secret pattern stripping.

Sentry.init({
  release: process.env.GIT_SHA,
  beforeSend(event) {
    // deny-list sensitive headers BEFORE the event leaves the process
    const headers = event.request?.headers;
    if (headers) { delete headers['authorization']; delete headers['cookie']; }
    // redact obvious PII / secrets from the body
    if (event.request?.data) event.request.data = redactSecrets(event.request.data);
    return event;
  },
});