Resource limits and OOM: cgroup ceilings, exit 137, and runtimes that ignore them
A container with no memory limit can OOM the whole node, and a runtime that ignores cgroup limits OOMs itself. --memory sets the cgroup ceiling, exit 137 is the OOM kill, and cgroup-aware runtimes (GOMEMLIMIT, MaxRAMPercentage) keep the heap under it.
The node went dark at 02:14. Not one container — the whole machine: kubelet unresponsive, SSH timing out, every pod on it flapping. The post-mortem found a single batch worker that loaded an unbounded result set into memory. It had no memory limit set, so its cgroup ceiling was the node’s entire RAM. The kernel’s OOM killer woke up, but with the host out of memory it started reaping by a heuristic score, and the highest-scoring victims were the fat JVM next door and, eventually, kubelet itself. One process with no limit took down a node hosting forty others. The fix was one line in the pod spec — a memory limit — and the deeper lesson was that the absence of a limit is not “unlimited, but fine”: it is “unlimited, and your blast radius is the whole host.”
The cgroup ceiling and exit 137
When you see exit code 137 in a production container, that number is telling you exactly what happened at the kernel level — and it is worth understanding before you chase a bug in your application code.
A container is a process in a cgroup, and --memory (Compose mem_limit, Kubernetes resources.limits.memory) writes the cgroup v2 memory.max ceiling. When the cgroup’s resident memory crosses that ceiling and cannot reclaim, the kernel OOM killer terminates a process inside that cgroup — usually PID 1, which ends the container. The container records exit code 137, which is 128 + 9: the process died from signal 9, SIGKILL, delivered by the OOM killer. That number is the single most useful diagnostic in production containers — 137 almost always means “hit the memory ceiling,” not a crash in your code.
docker run --memory=512m --memory-swap=512m myapp
# ^ equal to --memory disables swap for the container;
# omit it and Docker grants swap = 2× memory, so the
# container can quietly use 1 GiB before OOM and thrash first.
docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' myapp
# true 137 ← the kernel OOM-killed it at the cgroup ceilingThe contained case is the good one: with a limit, only your container dies, gets restarted by the orchestrator, and the node stays healthy. Without a limit (memory.max = max), the cgroup can grow until the host is out of memory, and then the system-wide OOM killer chooses victims across all cgroups by oom_score — which can be a different container, or a critical daemon. A missing limit does not protect your container; it endangers its neighbours. This is why production guidance is to always set memory limits, and to set requests equal to limits for memory (memory is incompressible — you cannot throttle it like CPU, you can only kill).
A container is OOM-killed and its state shows ExitCode 137, OOMKilled true. What does 137 specifically tell you?
Runtimes that ignore the cgroup limit
Setting the cgroup ceiling is only half the job — the runtime inside the container has to respect it. Many runtimes default to sizing their heap from the host’s memory, not the cgroup’s, because they read /proc/meminfo (the host total) rather than memory.max. A JVM before container-awareness would see a 64 GiB host, set its default max heap to ~16 GiB (¼ of host), and get OOM-killed the instant it tried to grow inside a 512 MiB cgroup — exit 137 on startup, before serving a request. Go’s garbage collector, left to defaults, only triggers when the heap doubles, so a service holding 400 MiB live can balloon past a 512 MiB ceiling between collections and get killed mid-request.
The fixes are runtime-specific knobs that tie the heap to the cgroup. Together they form a two-layer defence: the kernel limit is the hard stop, the runtime knob is the early warning that fires before you hit it — without both layers you are either flying blind or flying without a net.
# JVM: size the heap as a percentage of the cgroup limit (container-aware since JDK 10/8u191)
java -XX:MaxRAMPercentage=75.0 -jar app.jar # 75% of memory.max, not of host RAM
# Go (1.19+): a soft memory target the GC honours, set just under the cgroup ceiling
ENV GOMEMLIMIT=450MiB # for a 512 MiB limit — leave headroom for non-heap memory
# the GC runs more aggressively as the heap approaches GOMEMLIMIT instead of waiting to double
# Node: cap V8's old-space below the cgroup limit
node --max-old-space-size=400 server.js # MiB; V8 default is ~2 GiB regardless of cgroupThe pattern is the same across stacks: the cgroup limit is a hard wall enforced by the kernel with SIGKILL; the runtime knob is a soft target the runtime enforces by collecting garbage before it hits the wall. Set only the cgroup limit and the runtime cheerfully grows into the wall and dies; set only the runtime knob and a leak past it still has no kernel backstop. You need both — and you set the soft target a little below the hard wall, because non-heap memory (thread stacks, native buffers, metaspace) lives in the same cgroup and counts against memory.max too.
You set --memory=512m on a Go service but it still gets OOM-killed under load while reporting only ~400 MiB live heap. Why, and what knob fixes it?
- 01A container shows exit code 137 with OOMKilled true. Explain exactly what happened and why the number is 137.
- 02Why can a Go or JVM service be OOM-killed inside a 512 MiB cgroup even though its application data fits, and how do GOMEMLIMIT / MaxRAMPercentage prevent it?
A container is a process in a cgroup, and a memory limit writes that cgroup’s memory.max ceiling. When resident memory crosses it and cannot be reclaimed, the kernel OOM killer SIGKILLs a process inside the cgroup — usually PID 1 — and the container records exit 137 (128 + signal 9) with OOMKilled true, the single clearest signal in production that you hit the memory wall rather than crashing in code. Set —memory equal to —memory-swap to disable the container’s swap, otherwise Docker grants swap of twice the limit and the container thrashes before it dies. The reason to always set a limit is blast radius: with a limit only your container is killed and restarted while the node stays healthy; without one the cgroup ceiling is the whole host, and when the host runs out the OOM killer reaps across every cgroup by oom_score, potentially taking down a neighbour or kubelet itself. Setting the kernel ceiling is only half the job — the runtime inside must respect it, because JVM, Go, and Node default to sizing their heap from host RAM and will grow straight into the wall: a JVM picking ~16 GiB on a 64 GiB host dies on startup inside 512 MiB, and a Go service holding 400 MiB live can balloon past 512 MiB between garbage collections. Tie the heap to the cgroup with a soft target — MaxRAMPercentage for the JVM, GOMEMLIMIT for Go, max-old-space-size for Node — and set it a little below the hard wall, because non-heap memory shares the same cgroup and counts against memory.max too. The cgroup limit is the hard kernel backstop; the runtime knob is the soft target that collects before the wall. You need both. Now when you see a container cycling between exit 137 and restart, check two things in order: is a limit set at all, and does the runtime have a cgroup-aware soft target below it?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.