Alert quality and tuning
Most alerts are noise, and a wall of noise hides the one real intrusion. Senior tuning raises precision — killing low-value rules, scoping the rest — without quietly dropping the true positive you exist to catch.
The Target breach is the parable every detection engineer learns the hard way: the FireEye sensor did fire. The malware was named, the alert was raised, and it landed in a console that was already drowning. It sat next to thousands of other alerts that week — most of them benign, repetitive, ignorable — so the one that mattered was triaged as just more weather. Forty million cards later, the lesson wasn’t “buy a better sensor.” The sensor worked. The pipeline that was supposed to turn its output into a decision had been trained, alert by ignored alert, to assume everything was noise. That is alert fatigue, and it is not a human failing you can fix with a memo. It is a math problem: when 99% of what reaches an analyst is false, the analyst’s rational move is to stop reading.
By the end of this lesson you’ll know why most alerts are noise, how to tune detections for precision without silently dropping the one true positive, and how to measure whether your tuning helped or just hid the breach.
Why the wall of alerts is the actual vulnerability
A detection rule does not exist to fire; it exists to cause a correct action. Between the rule and the action sits a human with a finite attention budget, and that budget is the scarcest resource in the whole stack. Once you accept that, “we have 600 rules and a glowing dashboard” stops being reassurance and starts being a warning: every low-value rule you add spends down the attention you need for the high-value one.
The trap is the base rate. In a real environment the overwhelming majority of activity is legitimate. Suppose a rule flags 1 in 1,000 events as suspicious and your environment generates 10 million events a day — that’s 10,000 alerts. Now suppose the rule is genuinely good, with a 99% true-positive rate on actual attacks, but actual attacks are rare: maybe 5 real malicious events hidden in those 10 million. The rule catches all 5 (great recall) but also flags, say, 0.1% of the benign 10 million as suspicious — 10,000 false positives. Your analyst now faces 10,005 alerts to find 5 real ones. Precision is 5 / 10,005 ≈ 0.05%. Recall is perfect; the system is still useless, because no human reads 10,005 alerts looking for 5 needles before deciding the haystack isn’t worth searching.
This is the base-rate fallacy in operational form, and it is why “we caught it, the alert fired” is cold comfort. Coverage without precision doesn’t protect you — it buries you, and the breach hides in the rubble of your own detections.
Precision, recall, and the lever you actually pull
Two numbers govern every tuning decision, and conflating them is the most common senior mistake:
- Recall (sensitivity) — of the real attacks, what fraction did you catch? Missing a true attack is a false negative. This is the number that ends companies.
- Precision — of the alerts you fired, what fraction were real? Crying wolf is a false positive. This is the number that ends analysts — and a burned-out, desensitized analyst causes the false negative downstream.
The two are coupled by your threshold. Loosen a rule (fire on weaker signal) and recall rises but precision falls — more noise. Tighten it and precision rises but recall falls — you risk missing the real one. Tuning is not “make alerts go away.” Tuning is moving the operating point along that curve deliberately, with eyes open about which error you’re trading for which. The cardinal sin is tightening a rule until the noise stops and not noticing you tightened it past the true positive — buying a quiet console by going blind.
How a senior actually tunes: kill, scope, enrich, tier
You don’t tune by globally raising a threshold until it’s quiet. That’s the move that loses the true positive. You tune rule by rule, and the order matters:
- Kill the rules that can never cause an action. Every alert should map to a decision: if this fires, what would I do? A rule whose honest answer is “nothing, we always close it as benign” is pure tax — it spends attention and trains the team to auto-dismiss. Deleting it raises precision with zero recall cost, because it never caught a real attack to begin with. This is the cheapest precision you will ever buy.
- Scope, don’t silence. The vuln scanner that hammers your network at 02:00 trips your port-scan rule nightly. The wrong fix is to disable port-scan detection. The right fix is a scoped suppression: exclude that source IP and that window, and alert if the pattern appears from anywhere else or at any other time. You kept recall for the attack you care about and removed the noise from the source you trust. The danger of broad suppressions is that they rot — the trusted IP gets reassigned, the window drifts — so every suppression needs an owner and an expiry, or it becomes a permanent blind spot an attacker can hide inside.
- Enrich so the analyst decides in seconds, not minutes. A bare “failed login from 10.2.3.4” forces a five-minute investigation. The same alert enriched with whose account, geo/ASN, “this user normally logs in from Berlin, this is from a hosting provider in another country,” and a link to recent activity is a five-second decision. Enrichment raises effective precision: a true positive that was drowning in ambiguity is now obviously actionable.
- Tier by severity and route accordingly. Not every detection deserves a page at 03:00. Map detections to severity (ATT&CK tactic helps: a credential-dumping technique outranks a reconnaissance ping) and route low-severity, high-volume signals to a dashboard or a daily digest, reserving the pager for the handful that genuinely demand a human now. Tiering protects the attention budget without dropping recall — the low-signal events are still recorded, just not interrupting.
| Move | Effect on precision | Effect on recall | The failure mode to watch |
|---|---|---|---|
| Delete an always-benign rule | Up | Unchanged (never caught anything) | Deleting a rule that was catching real ones — confirm it never fired a true positive first |
| Globally raise the threshold | Up | Down — risks the true positive | Buying a quiet console by going blind to the real attack |
| Scoped suppression (source + window) | Up | Preserved for everything else | Suppression rots: stale IP/window becomes a permanent blind spot |
| Enrich the alert | Up (effective) | Unchanged | Enrichment source is slow/unavailable and stalls triage |
| Tier and route by severity | Up (per-channel) | Unchanged (still recorded) | A mis-tiered critical lands in a digest nobody reads |
▸Why this works
Why not just buy a model that scores everything and only surface the top 1%? Because a pure-precision optimizer with no recall floor will quietly learn to never alert on the rare thing it has the least data about — which is exactly the novel intrusion you most need to catch. Precision and recall are a constrained optimization: maximize precision subject to a recall floor on the techniques that would actually hurt you. That’s why tuning is anchored to a threat model (which ATT&CK techniques must I never miss?) and not to “reduce ticket volume” as a raw KPI. Optimize ticket volume alone and the cheapest path the system finds is to stop detecting the attacks that are hardest to detect.
Measuring whether the tuning worked — not just whether it’s quieter
A quieter console is not evidence of a better one; the Target console could have been made quieter by deleting the very rule that fired. So you measure the trade explicitly:
- Precision proxy — what fraction of fired alerts get actioned vs auto-closed as benign? A rule auto-closed 99% of the time is a precision problem you can see without waiting for an attack.
- Recall, deliberately probed — you cannot measure false negatives by waiting (you don’t know what you missed). You manufacture known true positives: purple-team exercises and detection unit tests that replay a real technique (per ATT&CK) and assert the rule fires. A tuning change that drops a purple-team detection is caught before an attacker exploits the same gap.
- Mean time to detect/triage — enrichment and tiering should pull this down; if MTTD climbs after a tuning pass, you traded clarity for quiet.
A high-volume port-scan rule fires every night because the internal vuln scanner at 10.0.0.9 runs at 02:00. Analysts have started ignoring all port-scan alerts. Pick the best tuning move.
A rule catches 100% of real attacks but, in a 10M-event/day environment, also fires 10,000 false positives a day against ~5 real events. Why is it still a failed detection?
You can't measure false negatives by waiting (you don't know what you missed). How does a senior team actually probe recall before an attacker does?
Order a precision-tuning pass on a noisy rule from safest-first to last, so you never drop the true positive:
- 1 Confirm the rule has never fired a real true positive (check history)
- 2 Identify the specific benign source/pattern causing the noise
- 3 Apply a scoped suppression (source + window) with an owner and expiry
- 4 Enrich the surviving alerts so the true positive is obviously actionable
- 5 Re-run the purple-team test to confirm the real technique still fires
- 01Explain, using the base rate, why a detection rule with perfect recall can still be a failed detection — and what that implies about coverage vs precision.
- 02Walk through how a senior tunes a noisy rule for precision without dropping the true positive, and how they prove they didn't.
Most security alerts are noise, and the breach hides not in a rule you forgot to write but behind the wall of false positives your existing rules generate — the base-rate fallacy in operational form, and the real lesson of Target, where the sensor fired into a console already trained to ignore it. Recall (did you catch the real attack?) and precision (were your alerts real?) are coupled by your threshold, and the cardinal sin is tightening a rule until it’s quiet without noticing you tightened it past the true positive. A senior tunes rule by rule: delete never-actioned rules (free precision), apply scoped suppressions with an owner and expiry instead of silencing whole detections, enrich alerts so a true positive is obviously actionable, and tier by severity so only the genuinely urgent few interrupt a human. And because you can’t measure false negatives by waiting, you probe recall deliberately — purple-team replays and detection unit tests that fire a known ATT&CK technique and assert the rule catches it. So next time someone celebrates a quieter dashboard, your first question is: quieter because we removed noise, or quieter because we just went blind to the one that matters?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.