Debugging distroless: no shell, no exec — bringing your own tools into another container's namespaces
A distroless image has no shell, so docker exec sh fails — nothing to exec into. Debug it from outside: an ephemeral debug container (docker debug / kubectl debug) sharing the target's namespaces with its own tools, or read the filesystem at /proc/PID/root.
The service shipped on gcr.io/distroless/static — 2 MB, no shell, no package manager, a clean security scan, everyone happy. Until it started returning truncated responses under load, and the on-call reached for the reflex: docker exec -it svc sh. OCI runtime exec failed: exec: "sh": executable file not found in $PATH. No sh. No bash. No ls, no cat, no curl — the image was just the binary. The instinct that has worked for a decade — get a shell inside and look around — hit a wall that the image was deliberately built to be. Worse, when the binary segfaulted ten minutes later, the container was gone, and with it any chance to inspect the filesystem it left behind. The team’s first reaction was to rebuild on a debug base and redeploy — which would have destroyed the very state they were trying to capture. The actual move was the opposite of going in: you bring your tools to the container from outside, joining its namespaces from a separate image that has a shell, without ever modifying the thing you are debugging.
Why exec fails: the shell was never there to find
docker exec sh does not summon a shell; it executes a binary that must already exist in the container’s filesystem. Distroless images (and scratch-based, and most well-hardened images) contain only the application binary and its runtime libraries — no /bin/sh, no busybox, no coreutils. That absence is the security feature: an attacker who achieves RCE finds no shell to pivot with, no curl to exfiltrate with, a near-empty attack surface. So exec: "sh": executable file not found is not a bug — it is the image working as designed, and the tool you reach for must respect that the running container stays unmodified.
The right primitive brings the tools instead of finding them. docker debug (Docker Desktop / Pro) attaches an ephemeral toolbox to a running container, giving you a shell-with-utilities in the target’s namespaces without changing its image or filesystem. In Kubernetes the same idea is kubectl debug -it <pod> --image=busybox --target=<container>: it spins up an ephemeral container in the pod that shares the target’s PID and network namespaces, so your busybox ps/curl/nc see the distroless process and its sockets directly. The --target is the part that matters — it joins the process namespace of the specific container, so ps shows the app’s PID 1, not just your debug shell. Nothing in the application image changes; you attach, look, detach, and the ephemeral container is discarded.
# Kubernetes: attach a busybox that shares the distroless container's namespaces
kubectl debug -it payments-xyz --image=busybox:1.36 \
--target=payments # join THIS container's PID + net namespace
# Inside: now you have tools the distroless image lacks
ps -ef # see the app's PID 1, shared PID namespace
wget -qO- localhost:8080/health # hit the app's port, shared net namespacedocker exec -it svc sh on a distroless container returns 'exec: sh: executable file not found in $PATH'. What is the correct interpretation and next move?
When the container is already dead: copy a static busybox, or read /proc/PID/root
Ephemeral debug containers need a running target to share namespaces with. If the distroless binary has already crashed, there is nothing left to attach to — the process and its namespaces are gone. Two senior moves cover the gaps:
- For a still-running container with no debug tooling available: copy a self-contained static binary in and run it, e.g.
docker cp busybox <ctr>:/busybox && docker exec <ctr> /busybox sh. Because busybox is statically linked, it needs no libraries from the distroless image — it carries everything. This does touch the container’s writable layer, so it is a deliberate, last-resort modification, not the clean ephemeral-container path. - To reach a container’s filesystem from the host: every running container’s root is mounted at
/proc/<host-PID>/rooton the host. WithPID=$(docker inspect --format '{{.State.Pid}}' svc),sudo ls /proc/$PID/root/applets you read the distroless filesystem — config, the binary, written files — using the host’s tools, with no shell in the container at all. This is how you inspect a shell-less image’s files without entering it.
Together, these two techniques cover everything a running shell-less container exposes: the ephemeral container sees the process and sockets; /proc/PID/root sees the filesystem. Without the ephemeral approach you face a forced choice between rebuilding the image (destroying state) or flying blind.
For a crash itself, debugging shifts left and out: capture the core dump (set --ulimit core=... and a host core_pattern, or have the runtime write the dump to a mounted volume) so a segfault leaves an artifact you analyze offline, and lean on the engine’s record — docker inspect State and docker events — since the dead container can no longer be entered by any means.
A distroless container has already crashed and is gone. Why won't an ephemeral debug container (kubectl debug --target) help here, and what does?
▸Why this works
Why is “bring tools from outside” the correct mental model rather than “get a shell in”? Because a hardened image’s whole value is that there are no tools inside it to find — for you or for an attacker. The debugging primitive that scales is therefore separation: the tools live in a throwaway image, the namespaces are shared so those tools see the real process and sockets, and the application image stays byte-for-byte unchanged. You never trade your security posture for your debuggability; you keep both by keeping them in different containers.
- 01Why does docker exec sh fail on a distroless image, and what is the correct way to get debugging tools against a running one?
- 02How do you inspect a shell-less container's filesystem from the host, and what changes once it has crashed?
A distroless image ships only the application binary and its runtime libraries — no /bin/sh, no busybox, no coreutils — so docker exec sh fails with ‘executable file not found’ because there is genuinely no shell to execute, and that absence is the point: an attacker with RCE finds nothing to pivot with. The correct debugging model is to bring tools from outside rather than find them inside, and never to rebuild the running image on a debug base, which would destroy the very state you are investigating. Against a running container, an ephemeral debug container — docker debug, or kubectl debug —image=busybox —target=<container> — attaches a throwaway toolbox that shares the target’s PID and network namespaces, so its tools see the distroless process’s PID 1 and sockets directly while the application image stays byte-for-byte unchanged. To read the filesystem without entering at all, every running container’s root is mounted on the host at /proc/<host-PID>/root, with the PID from docker inspect’s State.Pid. As a deliberate last resort you can docker cp a statically linked busybox into a running container and exec it, since a static binary needs nothing from the image — but this touches the writable layer. Everything above needs a live process: namespaces and /proc/<PID>/root exist only while the process holds them open. The moment a distroless container crashes, its namespaces are torn down and nothing can enter it, so post-mortem lives entirely in artifacts captured before death — a core dump from —ulimit core plus a host core_pattern or a dump on a mounted volume — and the engine’s own record in docker inspect State and docker events. You keep security and debuggability at once by keeping them in different containers. Now when you see ‘executable file not found’ on a hardened image, you will reach for kubectl debug —target rather than trying another shell name or rebuilding — and you will have already set up core dumps before the next segfault, not after.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.