docker logs and events: the rotation that eats your evidence and the daemon stream that kept it
docker logs replays whatever the logging driver captured from stdout/stderr — the default json-file driver's max-size and max-file silently rotate old lines away. docker events is the daemon's real-time stream: the OOM kill and restart no app log recorded.
The incident review wanted the logs from 02:00, when the worker started dropping jobs. The engineer ran docker logs worker --since 02:00 --until 02:30 and got back exactly nothing for that window — the oldest line was 02:47. Not a clock problem: the worker had logged a stack trace per dropped job at thousands of lines a second, and the default json-file driver was configured with max-size=10m, max-file=3. Thirty megabytes of ring buffer at that volume holds about ninety seconds. By the time anyone looked, the rotation had already chewed through the evidence and replaced it with newer noise. The root-cause line existed, was captured, and was deleted by the logging driver doing exactly what it was told. What did survive was in a place nobody had checked: docker events --since 02:00 still held the daemon’s own record — oom, then die, then start — the kernel killing the worker and the restart policy reviving it, a story the application logs could never have told because the app was dead before it could write it.
logs is a replay of the driver’s buffer, not a file you own
Why does this distinction matter at 2 AM during an incident? Because when you expect a file and get a ring buffer, you stop looking at the right time. docker logs does not read your application’s log file. It replays whatever the container’s logging driver captured from PID 1’s stdout and stderr. With the default json-file driver, every line is wrapped as JSON on disk under /var/lib/docker/containers/<id>/<id>-json.log, and — critically — that file is rotated by the driver’s own settings, which most teams never tune:
# Default driver: json-file. Rotation is OFF unless you set it,
# but production images and daemon.json usually set a small cap:
docker run --log-driver json-file \
--log-opt max-size=10m --log-opt max-file=3 worker
# => at most 3 files × 10m = 30m of history. A chatty app burns
# through that in seconds, and docker logs can never show more.
docker logs worker --since 02:00 --until 02:30 --timestamps
docker logs worker --tail 200 -f # follow the live tailThe trap is that the rotation is invisible from the docker logs side: there is no error, no gap marker — the oldest line is simply whatever survived, and --since for a window the driver has already discarded returns empty as if nothing happened. Two more facts that bite seniors: (1) only stdout/stderr is captured — an app logging to a file inside the container produces nothing for docker logs at all; and (2) some drivers are not even replayable — under journald or syslog you query the host’s journal, and under splunk/gelf docker logs may refuse entirely because the bytes left the host. The lesson: treat docker logs as a small, lossy ring buffer for live tailing, and ship logs off the node (driver → aggregator) for anything you must keep.
docker logs worker --since 02:00 --until 02:30 returns nothing, but the worker was definitely logging heavily then. The driver is json-file with max-size=10m, max-file=3. What happened to the evidence?
events is the daemon narrating itself in real time
When the application could not log the thing that killed it — because it was the engine or the kernel that acted — docker events is where the story lives. It is a live stream of every state transition the daemon performs, on containers, images, volumes, and networks, with --since/--until for replay over the daemon’s retained window:
docker events --since 02:00 --until 02:30 \
--filter container=worker --filter event=oom --filter event=die
# 02:14:09 container oom worker
# 02:14:09 container die worker (exitCode=137)
# 02:14:11 container start workerThe oom → die(137) → start triplet is the signature of a memory crash-loop, and none of it appears in docker logs — the app was SIGKILLed mid-breath, with no chance to write a farewell. Events surface what is otherwise invisible: OOM kills, health-status flips (health_status: unhealthy), restart-policy actions, kill/stop/destroy, and volume mounts. Pair it with docker stats for live cgroup memory/CPU and you can watch a container climb toward its limit and get reaped in real time. The mental model: logs is what the workload said; events is what the engine did to it — and on the worst nights only the second one was ever written down.
A container is crash-looping on memory but docker logs shows only a clean startup banner each cycle — no error, no stack trace. Where is the actual failure recorded?
▸Why this works
Why does Docker split the story across two streams instead of one log? Because they have different authors and different survival guarantees. logs is the workload narrating itself — rich, but only as durable as the driver’s ring buffer and only written while the process lives. events is the daemon narrating its actions on the container — sparse, but it captures the moments the workload cannot: the SIGKILL, the restart, the health flip. The split is the point: when the app dies mid-sentence, the daemon’s pen keeps moving.
- 01Why can docker logs return empty for a window when the app was definitely logging then, and what configuration controls it?
- 02What does docker events capture that docker logs structurally cannot, and why does it matter in a crash-loop?
docker logs is not a window onto your application’s log file — it replays whatever the container’s logging driver captured from PID 1’s stdout and stderr. With the default json-file driver those lines live in /var/lib/docker/containers/<id>/ and are rotated by max-size and max-file, so the entire history you can ever retrieve is max-file × max-size; a chatty app burns through a 30m buffer in seconds, and —since for an already-rotated window returns empty with no error and no gap marker — the evidence is gone and nothing tells you. Two further traps: an app that logs to a file inside the container gives docker logs nothing, since only stdout/stderr is captured, and drivers like journald, syslog, splunk, and gelf ship bytes off-host so docker logs queries elsewhere or refuses outright. So treat docker logs as a small, lossy ring buffer for live tailing and ship anything durable to an aggregator. When the thing that killed the container was the engine or the kernel, the app could not log it — and that is what docker events is for: the daemon’s real-time, —since-replayable stream of state transitions, where a memory crash-loop shows the oom → die(137) → start triplet that never reaches docker logs because the SIGKILLed app wrote no ending. logs is what the workload said; events is what the engine did to it. Pair events with docker stats to watch a container climb to its limit and get reaped in real time. Now when you see an empty —since window, you will check rotation settings before assuming the app never logged — and when a crash-loop leaves no error in logs, you will look in docker events first.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.