Observability
Structured logging: multiple-choice review
Crux Multiple-choice synthesis across the structured-logging unit — schema contracts, level routing, sampling economics, PII and injection defence, trace correlation, and audit separation.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one is a decision you make while a service is on fire at 03:00 — not a definition to recite, but a tradeoff between the bill, the alert, the audit, and the investigation.
Goal
Confirm you can connect the schema contract, the level ladder, sampling economics, PII and injection defence, trace correlation, and audit separation — the synthesis the individual lessons built toward.
Quiz
Completed
Two services on one team emit JSON logs, but one calls the field http.response.status_code and the other calls it status. Both are valid JSON. Why does this still break on-call?
Heads-up Both are structured: stable serialized records with addressable fields. The defect is schema drift, not a missing structure. Both are queryable; they are just not queryable by the same name.
Heads-up Regex-bridging drift defeats the point of structured logs and rots as new variants appear. The fix is a shared schema (OTel Semantic Conventions) enforced before code ships, ideally via a per-org wrapper logger.
Heads-up Backends index the field names they receive; they do not invent canonical mappings. status and http.response.status_code are two different indexed fields.
Quiz
Completed
A retry helper logs ERROR on every one of its 5 attempts, including the 3 that ultimately succeed. The on-call channel is now noise and a real ERROR was missed last week. What is the correct fix and why?
Heads-up Retry outcomes are operationally relevant for capacity and reliability and must stay visible in production; DEBUG is off in prod by default, so this hides them entirely instead of routing them correctly.
Heads-up Thresholds paper over a classification bug and will swallow genuine error bursts too. The level is the contract about what action is implied — fix the level, not the alert plumbing.
Heads-up Sampling ERROR silently discards failure forensics; you must keep 100% of WARN and ERROR. The problem is misclassification, not volume.
Quiz
Completed
A single service at 1000 req/s with 1 KB JSON logs is costing ~$260/month in ingest. What is the first lever to cut it ~90% without losing the ability to investigate failures?
Heads-up Uniform sampling loses 90% of ERROR lines and makes incident investigation impossible. Sampling must always preserve 100% of WARN and ERROR.
Heads-up Retention tiering controls how long you keep data, not how much you ingest per day. Ingest cost is volume times rate; sampling cuts the rate, retention only changes the storage horizon.
Heads-up Per-service sampling fragments the policy across the fleet with no central visibility and makes tail sampling impractical. Sampling belongs at the collector tier for centralised control.
Quiz
Completed
A handler interpolates user-submitted comment text directly into a log message string. A user submits a comment containing a newline followed by a forged ERROR JSON object. What class of failure is this, and what is the structural fix?
Heads-up The defect is structural — a newline acting as a record separator — not a character-encoding issue. UTF-8 normalisation does nothing to stop a forged log line.
Heads-up Forged log lines have caused real audit-trail manipulation and SIEM-rule bypass. Log message content is an input surface; Log4Shell was the extreme version of the same lesson.
Heads-up Blocklisting characters is fragile and misses JSON structural injection. The durable fix is to never interpolate input into the message string — pass it as a typed attribute and let the serializer escape it.
Quiz
Completed
A Node service reports that ~5% of log lines carry trace_id = '00000000000000000000000000000000'. How do you read this?
Heads-up A global misconfiguration would zero out close to 100% of lines, not 5%. A small, specific fraction points to particular async paths dropping the execution context.
Heads-up Sampling drops whole records, not individual fields within a record. The trace_id is missing because it was never populated, not removed downstream.
Heads-up Backends do not rewrite trace IDs. All-zeros is the emitted value when the logger finds no active context — a code-side async-propagation gap, not an ingest bug.
Quiz
Completed
A team stores operational logs and audit logs (logins, role grants, regulated-data access) in one index with a 30-day retention policy. Why is this a compliance failure, not just a tidiness issue?
Heads-up JSON is fine for audit logs — they share the schema and trace_id. The violation is the retention, immutability, and access properties of the shared index, not the encoding.
Heads-up Performance is incidental. The compliance failure is that audit events are being deleted before their mandated retention and are readable by too broad an audience.
Heads-up Audit logs use the same trace_id format; correlation still works. What differs is the backend's retention, immutability, and access tier — route via a category=audit attribute to a dedicated pipeline.
Recap
The unit’s through-line is one pipeline of decisions: a stable schema (OTel field names) makes logs queryable and joinable; the level ladder routes volume to storage and severity to humans; sampling keeps the bill proportional to incidents while always retaining WARN/ERROR; PII discipline and CWE-117 injection defence are first-class security concerns solved at the source and the collector; trace_id makes a log line a node in the observability graph instead of an island; and audit logs share the JSON schema but need their own pipeline, retention, immutability, and access. Every answer resolves back to treating the log line as an API contract your on-call and your auditor both depend on.