SIEM fundamentals
A SIEM ingests logs from every host, normalizes them into one schema, and correlates that firehose into a handful of alerts. The hard part isn't storage — it's turning millions of events an hour into the three that mean something.
3 a.m. An attacker logs in with a stolen password, the auth service emits a clean 200, and the load balancer, the database, the Linux box, and the Kubernetes audit log all dutifully write their lines — each in a different format, each in a different place. Every record needed to catch the intrusion already exists. The breach isn’t that nobody logged it; it’s that the proof was scattered across forty hosts in four formats, and nobody was watching the one sequence — new geo, then a failed MFA, then a sudo, then a 4 GB outbound transfer — that, read together, screams compromise. A SIEM exists to read those forty logs as one story.
By the end of this lesson you’ll know how a SIEM turns a firehose of raw logs from every host into a normalized, correlated stream — and why the genuinely hard part is alerting, not storage.
What a SIEM actually is
SIEM stands for Security Information and Event Management — but the acronym hides the real job. A SIEM is the place where every log in your estate is collected, made comparable, and watched for patterns no single log could reveal on its own. NIST SP 800-92, the canonical guide to log management, frames the core problem precisely: logs are generated by dozens of sources in incompatible formats and volumes that no human can read, so the value is entirely in centralizing and analyzing them, not in the act of writing them.
Think of it as a four-stage pipeline. Each stage is a place where real systems fail in production, so it’s worth walking them in order.
Stage 1–2: collection and normalization — the unglamorous half
Collection is the plumbing: ship every log to one place. Agents (a daemon on each host tailing files), syslog over the network, cloud provider APIs (CloudTrail, GCP Audit Logs), and direct integrations all feed the SIEM. The failure modes here are mundane and constant — a host whose agent died three weeks ago is a blind spot you don’t know you have, and a misconfigured firewall silently dropping UDP syslog loses records with no error anywhere. The senior reflex is to monitor the absence of logs: a source that suddenly goes quiet is itself an alert, because attackers turn logging off.
Normalization is where the real engineering lives. A Linux sshd line, an AWS CloudTrail JSON event, and an nginx access log all describe “someone did something from somewhere,” but they spell it completely differently. Normalization parses each format and maps its fields onto one common schema — so srcip, 192.168.1.5, and client_addr all become a single field like source.ip. Without this, correlation is impossible: you cannot join a failed SSH login to a suspicious database query if “the user” is acct, user_name, and principalId in three different logs.
| Raw source | Native field for “who” | Native field for “from where” | Normalized schema |
|---|---|---|---|
| Linux sshd | user | rhost | user.name / source.ip |
| AWS CloudTrail | userIdentity.principalId | sourceIPAddress | user.name / source.ip |
| nginx access | (none — derive from auth) | $remote_addr | user.name / source.ip |
| K8s audit | user.username | sourceIPs[] | user.name / source.ip |
Four sources, four schemas, one query language after normalization. This is why mature SIEMs lean on a shared field model — the Elastic Common Schema (ECS) and the OCSF (Open Cybersecurity Schema Framework) exist precisely so that source.ip means the same thing everywhere and a single rule can span every source.
Stage 3: correlation — reading logs as a story
Correlation is the SIEM’s reason to exist. A single failed login is noise. A failed login on a service account that has never failed, from a country you’ve never seen, followed within ninety seconds by a successful login and a privilege escalation, is an attack — and you can only see it by joining events across sources over a time window. That join is exactly what normalization made possible.
Correlation rules come in two broad shapes, and seniors mix them deliberately:
- Signature / known-bad rules map directly to attacker behavior. Frameworks like MITRE ATT&CK catalog techniques — say,
T1110Brute Force orT1078Valid Accounts — and you write rules that detect their telemetry: “10 failed auths then a success from one IP within 5 minutes” maps to credential stuffing. These are precise but only catch what you’ve named. - Anomaly / behavioral rules flag deviation from a baseline: a user pulling 50× their normal data volume, a server opening an outbound connection it has never made. These catch the unknown but pay for it in false positives, because “unusual” and “malicious” overlap only partially.
▸Why this works
Why not just alert on every individual suspicious line and skip correlation? Because volume destroys you. A mid-size estate emits millions of events an hour, and a meaningful fraction look mildly suspicious in isolation — every failed login, every new IP, every large download. Alert on each and you bury the real attack under ten thousand benign ones; within a week the on-call mutes the channel. Correlation is the act of raising the bar: it demands that several weak signals line up in a specific sequence and window before anything reaches a human, trading raw recall for the precision that keeps alerts credible.
Stage 4: alerting — where SIEMs live or die
The whole pipeline exists to produce a small number of high-fidelity alerts. This is the part teams underestimate. Storage is a solved, boring problem — you can buy more. Tuning is the eternal one. Every rule sits on a tradeoff between false positives (crying wolf until analysts stop listening — alert fatigue, the documented failure mode behind real breaches that were technically detected and then ignored) and false negatives (a real attack that never trips a rule). A rule tuned tight enough to be quiet may miss the novel attack; a rule loose enough to catch everything drowns the team. The senior measure of a SIEM is not how much it ingests but its signal-to-noise ratio: alerts per day a human can actually triage, and the fraction of those that are real.
Your SIEM fires 800 'suspicious login' alerts a day; analysts have quietly muted the channel and a real account-takeover slipped through last month. Pick the response that best fixes the root problem.
Why is normalization a prerequisite for correlation, rather than an optional nicety?
A SIEM ingests 40 million events/hour and stores them perfectly, but the on-call team has muted its alert channel. What's actually broken?
Order the four stages of the SIEM pipeline, from raw log to human:
- 1 Collect — ship every log to one place (agents, syslog, cloud APIs)
- 2 Normalize — parse each format into one shared schema
- 3 Correlate — join normalized events across sources over a time window
- 4 Alert — fire only high-fidelity matches to a human
- 01Walk through the four stages of a SIEM pipeline and name a real production failure at each.
- 02A SIEM ingests 40M events/hour flawlessly but a real account takeover was missed. Where's the problem and how do you fix it?
A SIEM — Security Information and Event Management — exists because every record needed to catch an intrusion already exists, just scattered across dozens of hosts in incompatible formats with nobody reading them as one story. It works as a four-stage pipeline. Collection ships every log to one place (agents, syslog, cloud APIs) and treats a source going silent as its own alert. Normalization parses each format onto a shared schema (ECS/OCSF) so that source.ip means the same thing in an sshd line, a CloudTrail event, and a K8s audit log — without which correlation is impossible. Correlation joins those normalized events across sources over a time window, mixing signature rules mapped to MITRE ATT&CK techniques with anomaly rules against a baseline, so a sequence of weak signals becomes one detection. Alerting emits only the high-fidelity matches to a human, and this is where SIEMs live or die: storage is easy, but the eternal tradeoff between false positives (alert fatigue) and false negatives (a missed attack) makes signal-to-noise the real senior measure. Next time you see a SIEM praised for ingesting 40 million events an hour, your first question is the right one: how many alerts a day does a human actually act on, and how many of those are real?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.