docker exec and inspect: joining a container's namespaces and reading the engine's ground truth
docker exec joins a running container's namespaces to give you a shell, but it runs as a sibling of PID 1 — not as PID 1 — so a healthy shell can hide a dead app. inspect reads the engine's truth: State.ExitCode where 137 is OOM-kill and 143 is SIGTERM.
The pager fired at 03:12: the API was returning 502s, but the container showed Up 3 hours and the readiness probe was green. On-call did the obvious thing — docker exec -it api sh, then curl localhost:8080/health. Connection refused. So the app was down. But ps inside that shell showed the Java process running, PID 1, eating 100% of one core. The app was not crashed; it was deadlocked in a GC death spiral, alive enough to hold PID 1 open so the container never exited, dead enough to serve nothing. The shell had lied by omission: exec had dropped them into a brand-new process that was a sibling of the app, sharing its namespaces but not its fate. The real signal was one command away — docker inspect would have shown the OOMKilled flag flickering and the restart count climbing — but the shell felt like being inside the app, and that feeling cost twenty minutes.
exec joins namespaces — it does not become the app
A container is not a box; it is a normal host process wrapped in namespaces (PID, mount, net, IPC, UTS) and cgroups. docker exec asks the engine to spawn a new process inside those same namespaces: same network stack, same filesystem view, same PID-space — but it is a fresh process, a child of the container’s PID 1, not PID 1 itself. This is exactly what nsenter --target <host-PID> --all does by hand when you bypass Docker: it setns()-es into every namespace of the target. The two are the same mechanism; docker exec is the engine-mediated, cgroup-correct version, while nsenter --target <pid> --all is the raw kernel call you reach for when the engine is wedged or the container has no entry the daemon will honor.
# Engine-mediated: join api's namespaces, land as a child of its PID 1
docker exec -it api sh
# Raw kernel: find the host PID, setns into ALL of its namespaces
PID=$(docker inspect --format '{{.State.Pid}}' api)
sudo nsenter --target "$PID" --all # --mount --uts --ipc --net --pidThe consequence that bites: your shell is not PID 1. If the application is hung, exec still succeeds — you are a separate process. If the application has its own internal threads stuck, your ps sees them but your shell keeps responding. A healthy prompt proves the namespace is alive, not that the workload is. The honest checks from inside are: ps -ef to confirm PID 1 is the process you expect and is not zombie/defunct, and probing the app’s own port (curl localhost:PORT) rather than trusting that “exec worked, so the app works.”
You run docker exec -it api sh and get a working shell, but curl localhost:8080 inside it is refused. What has the working shell actually proven?
inspect is the engine’s ground truth — and the exit code is a sentence
When the workload is the question, stop trusting the inside view and read what the engine recorded. docker inspect returns the daemon’s structured state, and .State.ExitCode is the most compressed incident report you will get:
docker inspect --format '{{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}}' api
# exited exit=137 oom=true restarts=14Exit codes follow 128 + signal: 137 = 128 + 9 (SIGKILL) — almost always the OOM-killer when OOMKilled:true, or a docker kill; 143 = 128 + 15 (SIGTERM) — a clean stop request the app honored (or docker stop’s grace timer firing). Reading 137 as “the app crashed with a bug” is the classic misdiagnosis: nothing in the app’s code threw — the kernel cgroup memory controller killed it for breaching its limit, and the fix is a memory limit or a leak hunt, not a stack trace. A climbing RestartCount with oom=true is a crash-loop on memory, not a logic bug. inspect also hands you .State.Pid (the host PID, the bridge to nsenter), the resolved mounts, the effective env, and the health-check log — the facts that survive after the container is gone.
docker inspect shows State.ExitCode 137 with OOMKilled true and RestartCount 14. What is the correct read?
▸Why this works
Why trust inspect over the shell? Because the shell is a live view of a moving system and inspect is the engine’s recorded verdict. The shell can only show you the present instant — and if the app already died and the restart policy revived it, the shell shows a fresh, healthy process while inspect’s RestartCount and last ExitCode preserve the crash you are actually chasing. Ground truth is the layer that remembers; the inside view forgets the moment it restarts.
- 01Why can docker exec give you a perfectly healthy shell while the application inside is dead, and what is the relationship to nsenter?
- 02How do you read docker inspect's State.ExitCode, and why is 137 routinely misdiagnosed?
A container is an ordinary host process fenced off by namespaces and cgroups, and docker exec spawns a new process inside those same namespaces — the engine-mediated equivalent of nsenter —target <host-PID> —all, which setns()-es into the lot by hand. The catch every senior eventually learns the hard way: that new process is a sibling of the container’s PID 1, not PID 1 itself, so exec succeeds and your shell stays responsive even when the application is deadlocked or spinning. A green prompt proves the namespace is alive, never the workload; the honest checks from inside are ps -ef to confirm PID 1 is the process you expect and probing the app’s actual port. When the question turns from “what is it doing” to “why did it die,” stop trusting the live view and read docker inspect — the engine’s recorded verdict. State.ExitCode speaks in 128 + signal: 137 is an external SIGKILL (with OOMKilled:true, the kernel memory-killer, a limit breach, a crash-loop — not an app bug and not a stack-trace problem), while 143 is a graceful SIGTERM the app honored. docker stop walks SIGTERM then SIGKILL, so 143 versus 137 tells you whether the shutdown was clean or forced. A rising RestartCount with OOMKilled:true is a memory crash-loop, and because the live shell forgets the moment the restart policy revives the container, inspect is the one view that remembers the crash you are chasing. Reach for raw nsenter via .State.Pid only when the daemon itself is wedged. Now when you see a green shell next to a refused port, you know to reach for inspect before trusting the prompt — and to read ExitCode 137 as a memory limit problem, not a code bug.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.