Cloud logging and trails

What the audit trail actually records

There are two log planes in any cloud, and people constantly confuse them. Data-plane logs record traffic through your services: HTTP requests to a load balancer, reads on an object, queries to a database. Control-plane logs — AWS CloudTrail, GCP Cloud Audit Logs, Azure Activity Log — record management actions against the cloud API itself: who called CreateUser, who attached a policy, who changed a security group, who rotated a key, who deleted a bucket. When you are reconstructing a breach, the control-plane trail is the one that matters, because every privilege escalation and every act of persistence is, underneath, a sequence of authenticated API calls.

A single CloudTrail event is a structured record with the fields an investigator actually needs: eventTime, eventName (the API called), userIdentity (the IAM principal, including assumed-role session and the originating account), sourceIPAddress, userAgent, the request parameters, and the response. Read those five fields across a few thousand events and you can answer the only question that matters at 02:00: which identity did this, when, from where, and what did it touch. This is why the trail is called the source of truth — it is the one place where the cloud itself, not your application, attests to what happened.

The catch is default coverage. Out of the box, a trail typically logs management events but not data events — an attacker reading every object out of an S3 bucket generates zero control-plane records unless you explicitly enabled data-event logging, which is off by default partly because it is voluminous and billed per event. The senior reflex: know precisely what your trail does and does not capture before the incident, because “we assumed S3 GetObject was logged” is a discovery you do not want to make while exfiltration is in progress.

Why the trail is the attacker’s first target

Think like the adversary for one paragraph. You have compromised a credential. Your goals are escalate, persist, and act — mine crypto, exfiltrate data, ransom the account. Every one of those leaves API-call evidence in the trail. So the highest-leverage move you can make, before anything noisy, is to blind the defender: stop the trail, delete it, or stop it from delivering to its destination. This is not theoretical — “defense evasion via impairing logging” is a named, catalogued technique in the MITRE ATT&CK cloud matrix precisely because it is the standard opening move. A trail an attacker can switch off is a smoke detector with an accessible off switch: it protects you exactly until the moment you need it.

Tamper-resistance: turning a log into evidence

The difference between a log and evidence is whether the person being recorded could have altered it. Four controls, layered, get you there.

Deliver to immutable, append-only storage. The trail should write to object storage with an object-lock / retention policy in compliance mode, so that even an account administrator cannot delete or overwrite an object before its retention period expires. Append-only is the property that defeats “edit history” — you can add records, never rewrite them.

Put the sink in a separate account. This is the single highest-value architectural move. A compromised principal in the workload account should not have IAM permissions over the log archive at all. Centralize trails from every account into one dedicated, locked-down logging account whose only job is to receive and retain. Now blinding the trail requires compromising a second, far smaller, far more tightly controlled account — you have raised the cost of evasion by an order of magnitude.

Verify integrity cryptographically. Enable log-file integrity validation (CloudTrail signs each delivered file and emits a hash-chained digest). At investigation time you can prove a file was not modified after delivery — turning “we think this is complete” into “we can demonstrate this is complete,” which is what makes the trail hold up in a forensic or legal context.

Alert on the act of tampering itself. The API calls that disable logging — StopLogging, DeleteTrail, PutEventSelectors narrowing scope, deleting a log sink — are themselves logged events. Wire a real-time alert on them. The attacker’s first move becomes their loudest. This is also the answer to NIST SP 800-137’s continuous-monitoring premise: a control you set once and never watch has unknown status; security state has to be observed on an ongoing basis, and “is logging still on?” is the most basic thing to observe.

Reading the trail well, not just keeping it

Tamper-resistance keeps the record honest; it does not make it useful on its own. A trail you never query is just expensive storage. Two senior habits close the gap. First, centralize and normalize: ship every account’s trail into one queryable store so an investigator runs a single query across the whole org, not twelve. Second, baseline the normal so the abnormal stands out: a new IAM user created at 03:00, a console login from a country you don’t operate in, a sudden burst of RunInstances in an unused region — these are signal only if you know what boring looks like. This is exactly the loop NIST SP 800-137 calls continuous monitoring: collect, assess against a known baseline, respond, repeat — security status is a thing you keep measuring, not a box you checked at launch.