Containers for Go: GOMAXPROCS vs CPU quota, GOMEMLIMIT vs OOM, and the multi-stage Dockerfile that caches right

The CPU lie: GOMAXPROCS vs the CFS quota

GOMAXPROCS defaults to the number of CPUs the OS reports — and in a container that is the host, because a cgroup CPU limit is not a number of cores. limits.cpu: 2 becomes cpu.max: 200000 100000: 200 ms of CPU time per 100 ms accounting period, summed across all threads. The two numbers live in different units, and the runtime historically read only the wrong one. The failure mechanics on a 64-core node: GOMAXPROCS=64 means up to 64 OS threads executing goroutines in parallel, plus GC background workers sized at a quarter of GOMAXPROCS — 16 more threads that wake together during collection. Under load they exhaust the 200 ms quota a few milliseconds into each period, and the scheduler hard-stops the entire cgroup until the next one. Latency does not degrade smoothly; it quantizes into multiples of the throttle window, which is exactly the 8x p99 cliff from the Hook. Diagnosis is two reads: /sys/fs/cgroup/cpu.stat (nr_throttled climbing against nr_periods, throttled_usec enormous) and the runtime metric for GOMAXPROCS showing the host core count.

The fix ladder, oldest to newest: set the GOMAXPROCS env var in the pod spec next to the CPU limit so they move together; or import go.uber.org/automaxprocs, which reads the cgroup quota at startup; or run Go 1.25+, where the runtime finally does it itself — on Linux the default GOMAXPROCS now respects the cgroup CPU bandwidth limit (rounded up, never below 2, still capped by host CPU count) and the runtime even re-checks it as limits change. The honest caveats: only limits count — CPU requests set cpu.weight, which is a contention ratio, not a quota, and no GOMAXPROCS value can be derived from it; an explicit env var or runtime call always wins; and pods that deliberately run limit-less still default to host CPUs, which is the correct behavior for them.

Memory: the OOM-kill vs GC-pressure tradeoff

The memory limit has the same shape: the kernel enforces limits.memory by OOM-killing the container, and the Go heap grows by a policy that has never heard of cgroups. With default GOGC=100, peak heap reaches roughly twice the live set — a service with 600 MiB live in a 1 GiB pod is not close to its limit, it is past it at the next GC cycle boundary. GOMEMLIMIT (Go 1.19+) closes the loop: set it to about 90% of the container limit and the GC pacer treats it as a soft ceiling for all runtime-managed memory — heap, stacks, runtime structures. The mechanism is the part worth keeping: as usage approaches the limit, the pacer schedules collections increasingly aggressively, converting memory pressure into GC CPU time instead of a SIGKILL. The tradeoff is explicit and usually correct — a service that burns 30% CPU in GC during a peak is degraded; a service the kernel killed is down, and it lost every in-flight request on the way out.

Two design details keep this honest. The limit is deliberately soft: the runtime caps GC CPU at roughly 50%, so when live heap genuinely will not fit, Go exceeds the limit (and may then be OOM-killed) rather than livelocking in back-to-back collections — the death-spiral guard. And the 90% headroom is not superstition: cgo allocations, mmap regions, and kernel memory charged to the cgroup are all invisible to the runtime, so GOMEMLIMIT=limit means the pod dies while Go believes it is fine. Set it as an env var in the deployment, next to the memory limit, so the two numbers are reviewed and changed together — env-driven config is the contract here, the same spiral as the configuration lesson in go/07.

The Dockerfile that caches right

The static binary turns the container story into a packaging exercise — the deploy artifact is one file, and the image exists to deliver it plus a TLS trust store:

FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download            # its own layer: busts only when deps change
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/api /api
ENTRYPOINT ["/api"]

The layer order is the caching strategy. A layer rebuilds when its inputs change, and every layer after it rebuilds too — so the question for each instruction is how often its inputs change. go.mod and go.sum change rarely; source changes every commit. Copying only the module manifests, downloading, and then copying source means an ordinary commit replays the download layer from cache (0 s) and pays only for the compile; invert the order — COPY . . first — and every commit re-downloads the entire module graph (30-90 s on a real dependency tree, every build, forever). BuildKit cache mounts go one step further (RUN --mount=type=cache,target=/go/pkg/mod go mod download keeps the module cache across builds even when go.mod changes), but the COPY ordering is the part people get wrong.

The runtime stage choice is a real decision. scratch is the minimum — the image is literally your binary — but you inherit the missing pieces: no CA certificates (first HTTPS call fails), no tzdata, no /etc/passwd to run as non-root, no shell for debugging. distroless/static costs about 2 MiB extra and ships exactly those: CA roots, tzdata, a nonroot user — still no shell, which is a security feature and a debugging tradeoff you cover with ephemeral debug containers. Either way the final image lands around 10-15 MiB against the builder image at roughly 850 MiB — which must never ship. Numbers worth keeping: binary ~7-9 MiB stripped, distroless/static base ~2 MiB, pull time effectively the registry round trip.