Static binaries and cross-compilation: CGO_ENABLED=0, the missing ld-linux mystery, and build-flag discipline
CGO_ENABLED=0 removes libc: no ld-linux interpreter, pure-Go DNS resolver, scratch-safe binary. GOOS/GOARCH cross-compiles without a C toolchain until cgo enters. -trimpath makes builds reproducible, -X injects versions, go version -m audits the provenance of any binary.
The deploy hit the new scratch image at six in the evening and the pod died instantly: exec /server: no such file or directory. The binary was right there — a debug container showed it, 9.1 MB, executable bit set. The team chased phantom COPY bugs for an hour before someone ran file on it: dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2. The error had never been about the binary. execve found the file, read its ELF header, and went looking for the interpreter the header named — and scratch contains nothing, ld-linux included. The dependency came from a default nobody had questioned: plain go build on the glibc CI runner had CGO_ENABLED=1, and importing net was enough to link libc for DNS. CGO_ENABLED=0 shipped that night. Two weeks later, act two: a hostname that had always resolved stopped resolving — the old image had been honoring an nsswitch.conf with LDAP-backed hosts via the libc resolver; the pure-Go resolver reads /etc/hosts and resolv.conf, period. Static linking is not a packaging detail. It swaps your DNS resolver.
The fork in every build: CGO_ENABLED
$ CGO_ENABLED=1 go build -o server-cgo . # default when building natively with a C toolchain
$ CGO_ENABLED=0 go build -o server-static . # pure Go, no C anywhere
$ file server-cgo
server-cgo: ELF 64-bit LSB executable, dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2 ...
$ file server-static
server-static: ELF 64-bit LSB executable, statically linked ...
$ ldd server-cgo
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6Why does a language with its own runtime link libc at all? Two stdlib packages reach into C when cgo is available: net (hostname resolution via getaddrinfo) and os/user (account lookups). With CGO_ENABLED=1 — the default whenever you build natively and a C compiler is present — those paths compile in, and the linker records a dynamic interpreter in the ELF header. That is the whole anatomy of the Hook mystery: execve opens the binary, reads PT_INTERP, and tries to load /lib64/ld-linux-x86-64.so.2 — a file that does not exist in scratch. The ENOENT it returns names the binary you ran, not the interpreter it failed to find, which is why the error reads like a lie. The Alpine flavor is the same bug with different paint: a glibc-built binary asks for the glibc loader, but musl ships /lib/ld-musl-x86_64.so.1.
The resolver swap is the subtler half. With cgo compiled in, Go decides per lookup whether to call getaddrinfo or its own DNS client, based on what it finds in /etc/nsswitch.conf and /etc/resolv.conf plus a few env vars — anything Go cannot emulate (mdns or nis modules, exotic resolv options) routes through libc. CGO_ENABLED=0 removes the choice: always the pure resolver, which reads /etc/hosts and /etc/resolv.conf and speaks DNS itself. In scratch there is no nsswitch.conf at all, so the Go resolver assumes the standard files-then-dns ordering — fine, until your infra silently depended on an nsswitch plugin: LDAP-backed hosts, mDNS, nscd caching all vanish without an error message. When the decision itself is the suspect, GODEBUG=netdns=go+1 forces the pure resolver and logs the choice; netdns=cgo+1 forces the other branch.
A Go service is copied into a scratch image. At startup the container dies with exec /server: no such file or directory — yet the file is present and executable. What actually failed?
When you cannot drop cgo — and the cross-compile matrix
The honest exceptions list is short. mattn/go-sqlite3 wraps the C library — modernc.org/sqlite is the pure-Go escape hatch, at a real (benchmark-it-yourself, often 1.5-2x) speed cost. The race detector is built on the C++ ThreadSanitizer runtime, so -race builds require cgo — which is fine, because that is a CI and test artifact, never the release binary. Some corporate DNS environments genuinely need nsswitch modules, which means the cgo resolver. And the plugin package. The working pattern: cgo where the C dependency actually lives, CGO_ENABLED=0 for the deploy artifact.
Cross-compilation is where Go collects the payoff for all of this:
$ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o server-arm64 ./cmd/server
$ go tool dist list | wc -l # the supported GOOS/GOARCH matrix
44Two env vars from any dev machine — a Mac laptop emits Linux ARM binaries with no extra installation, because the Go toolchain ships code generation for every target and the pure runtime makes syscalls directly, with no per-target libc to find. Contrast the C world: a cross toolchain, a sysroot, and a build system that respects both, per target. The moment cgo enters, you inherit exactly that zoo — cross-compiling cgo code needs a real C cross-compiler (CC=aarch64-linux-gnu-gcc, or the modern zig cc trick) and every transitively linked C library built for the target.
▸Why this works
Why can Go cross-compile so casually when C cannot? Because the hard part of cross-compilation was never code generation — compilers have emitted foreign instruction sets forever. The hard part is the target userland: headers, libc, the link-time environment. Pure Go carries its own runtime, performs syscalls itself, and links statically, so there is no target userland to reproduce. CGO_ENABLED=1 reintroduces the C userland, and with it the entire problem Go had deleted.
Flag discipline and provenance
When you ship a binary, you are making three implicit claims: this code came from this commit, this is the version the on-call will read in the alert, and this image is identical to what CI verified. None of those claims are true by default — they require three flags and one audit command.
$ go build -trimpath \
-ldflags="-s -w -X main.version=v1.42.0" \
-o server ./cmd/server
$ go version -m server
server: go1.25.4
path example.com/api/cmd/server
mod example.com/api v1.42.0
dep github.com/jackc/pgx/v5 v5.7.1 h1:...
build -trimpath=true
build CGO_ENABLED=0
build vcs.revision=9f3c2e1a...
build vcs.modified=false-trimpath strips the build machine paths from the binary, so the same source and toolchain produce a bit-identical artifact on any machine — the precondition for reproducible builds and for trusting that a binary corresponds to a commit. -ldflags="-s -w" drops the ELF symbol table and DWARF debug info, typically a 25-30% size cut (a 9 MB service lands near 6.5 MB). The honest accounting: panic stack traces still symbolize, because the runtime keeps its own pclntab — what you lose is DWARF, meaning delve against the production binary, gdb, and external symbolization tools. The senior move is to stop treating it as a tradeoff: build stripped for deploy and archive the unstripped twin (or upload symbols to your error tracker), so you get the small image and the debuggable artifact. -X importpath.var=value injects the human-facing version into a string variable at link time; since Go 1.18 the toolchain embeds vcs.revision, vcs.time, and vcs.modified automatically, so the commit is recorded even when nobody passed -X.
The audit superpower is the last line: go version -m works on any Go binary — including one copied out of a production container — and prints its module, full dependency list with hashes, and build settings. The 3 a.m. question of what exactly is running stops being archaeology and becomes one command.
You ship release binaries built with -ldflags=-s -w to cut size. A production panic comes in. What does the stack trace look like?
- 01Dissect the exec /server: no such file or directory failure in scratch — what did execve actually look for, and which default created the dependency?
- 02What is the release build-flags discipline for a Go service, and what does each flag honestly cost?
Every Go build forks on CGO_ENABLED. With cgo on — the native default — net and os/user link libc, the ELF header records a dynamic interpreter, and your image must supply ld-linux and glibc or die at execve with an ENOENT that names the wrong file. With CGO_ENABLED=0 the binary is genuinely static: one file, scratch-safe, and the DNS resolver is the pure-Go one that reads /etc/hosts and resolv.conf instead of consulting nsswitch.conf through getaddrinfo — a behavioral change, not just a linking change, and the reason DNS can differ between your old debian image and your new scratch one. GODEBUG=netdns settles per-lookup resolver questions. Cgo stays where C genuinely lives: sqlite via mattn (or pay the pure-Go speed tax with modernc), race-detector builds in CI, nsswitch-dependent environments. Cross-compilation is the dividend: GOOS and GOARCH select from about forty targets with no extra toolchain, because pure Go carries its runtime and syscalls with it — until cgo re-imports the C userland and its cross-toolchain zoo. Release flags are a discipline, not folklore: -trimpath for reproducible, provenance-grade builds; -X for the semver humans read; -s -w for a quarter to a third off the binary, paid for in DWARF — panics still symbolize via pclntab, but keep an unstripped twin if you ever attach a debugger. And go version -m reads module, deps, hashes, and the embedded VCS revision off any binary you can copy out of a container — the audit tool you want on the worst day. Now when you pull a mystery binary out of a production container at 3 a.m., you know the exact command to answer what, when, and from which commit it came.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.