Forensics basics

Order of volatility: collect the fragile stuff first

The single most important rule in evidence collection is order of volatility: capture the data that disappears fastest, first. RAM is gone the instant the box reboots or the pod is rescheduled. Network connection state, running processes, and open file handles vanish within seconds to minutes. Disk survives a reboot; an off-site log or backup survives the disk. So you collect up the volatility ladder — memory before disk, live state before stored state — because every minute you spend on the durable stuff is a minute the fragile stuff is decaying.

This is exactly why “just SSH in and look around” is so destructive. Reading files updates access timestamps. Running tools allocates memory, overwriting the unallocated RAM where deleted-but-not-yet-reclaimed artifacts (an attacker’s in-memory payload, a freed buffer with a credential) still live. Every command is an observer effect: the act of looking changes what you can later prove. The senior move is the opposite reflex — acquire before you analyze. Snapshot the volatile state to somewhere else, then do your poking on the copy.

In cloud and Kubernetes the rule still holds, the mechanics just shift. You don’t image a physical disk; you snapshot the EBS volume, dump the container’s memory via the node before the pod is killed, and — critically — set the compromised instance’s auto-scaling and restart policy to not terminate it, because a rescheduled pod is a wiped crime scene. Cordon the node, snapshot, then investigate the snapshot.

Chain of custody: evidence you can defend

An artifact is only as trustworthy as the record of who touched it and when. Chain of custody is the unbroken, documented trail proving an artifact wasn’t altered between collection and conclusion: who collected it, when, with what tool, where it was stored, and every hand it passed through. The technical anchor is a cryptographic hash — you SHA-256 the memory dump or disk image at acquisition, write that hash down, and anyone can re-hash the copy later and confirm it matches bit-for-bit. A broken hash means the evidence was modified; a matching hash means it’s pristine.

The discipline that protects the hash is work on copies, never originals. Image the disk, hash the image, then do all analysis against a mounted read-only copy. The original (the snapshot, the sealed image) never gets touched again. This isn’t bureaucratic theater — if the incident becomes a legal matter, a regulatory disclosure, or even just an internal dispute about blame, evidence with a broken or missing custody chain is worthless. You’ll know what happened and be unable to prove it, which in a breach-notification or insurance context is the same as not knowing.

IOCs: the fingerprints you search for and share

An Indicator of Compromise (IOC) is an observable artifact that signals an intrusion: a malicious IP or domain, a file hash, a suspicious registry key or cron entry, a user-agent string, a mutex name. IOCs do two jobs. First, scoping: once you have one — say the SHA-256 of a dropped binary — you sweep every host for that hash to find the full blast radius, not just the box that paged you. Second, sharing: IOCs are the lingua franca of threat intel, so an IP you extract can be fed to detection rules and shared with peers.

The senior caveat: IOCs sit at the bottom of the Pyramid of Pain. Hashes and IPs are trivial for an attacker to change — a recompile gives a new hash, a new VPS gives a new IP. They’re cheap to collect and cheap to evade. The durable signal is higher up: TTPs — the tactics, techniques, and procedures the adversary uses, the thing MITRE ATT&CK catalogs. Mapping your findings to an ATT&CK technique (say T1059, command-and-scripting interpreter) survives the attacker swapping infrastructure, because changing how they operate is genuinely expensive. So extract IOCs to scope this incident, but reach for the behavior to build detection that catches the next one.

The timeline: turning artifacts into a story

A pile of artifacts isn’t an answer. The deliverable of forensics is a timeline: a single, time-ordered sequence that fuses events from every source — auth logs, process-creation events, file modification times (the MAC times: modified, accessed, changed/created), network connections, deploy logs — into one narrative of what the attacker did, in order. This is where supertimelining tools earn their keep: they normalize timestamps from dozens of artifact types onto one axis so you can read the intrusion top to bottom.

The two hard problems are time and gaps. Time: correlating across sources demands a common reference — normalize everything to UTC, and watch for clock skew between hosts and for an attacker who tampered with timestamps to hide (timestomping; a file whose modified time predates its inode-change time is a tell). Gaps: the absence of a log is itself evidence — a deleted shell-history file or a window where audit logging was off tells you where the attacker worked and that they tried to cover it. A good timeline doesn’t just say what happened; lined up against MITRE ATT&CK it shows the progression — initial access, execution, persistence, lateral movement — and that progression is what tells you whether you’ve actually scoped the whole intrusion or only seen its tail.