Core dumps and post-mortem: ulimit, core_pattern, build-id matching, and unwinding a SIGSEGV after the fact
A core dump freezes a crashed process for offline autopsy, but only if ulimit -c and core_pattern allow it. Dumps reach gigabytes. gdb matches the dump to symbols by build-id; a mismatched build gives a garbage stack — the classic works-in-dev, cores-in-prod trap.
A payments service crashed once a week with a SIGSEGV no one could reproduce, and for two months the team had nothing but a one-line log. The first fix was not in the code — it was discovering that ulimit -c was 0 in the container, so the kernel had been throwing every core dump away. They set it, and the next crash dropped a 6 GB core. gdb opened it and printed a stack that was pure garbage: ?? frames, addresses with no symbols, a backtrace that pointed nowhere. The cause was the second trap: the binary in the running container had been rebuilt and redeployed since the engineer’s local copy, so the build-ids did not match, and gdb was unwinding the dump against the wrong symbol table. Once they pulled the exact binary by its build-id from the registry, the same 6 GB of frozen memory unwound into a clean stack ending in a use-after-free in the retry path — the bug that had been invisible for two months, sitting in the dump the whole time.
Generating a core: ulimit, core_pattern, coredumpctl
Before you can autopsy a crash, you have to answer a harder question: did the kernel even save it? Most teams first learn about core capture at the worst possible moment — when the bug they needed is gone.
A core dump is the kernel writing the crashing process’s memory image — address space, registers, the stack of every thread — to a file so you can autopsy it after the process is gone. But three gates decide whether you get one. First, the per-process resource limit ulimit -c: if it is 0 (the default in many container base images and systemd units), the kernel silently discards the dump. Second, /proc/sys/kernel/core_pattern decides where it goes — a literal path template like core.%p.%e, or, if it starts with |, a pipe to a handler program. On modern systemd hosts core_pattern pipes to systemd-coredump, and you retrieve dumps with coredumpctl list and coredumpctl gdb <pid> instead of hunting for a file. Third, in containers the limit and pattern live on the host kernel namespace, so a core_pattern that pipes to a handler not present in the container’s mount namespace fails — a common reason “I set ulimit and still got nothing.”
The size is not small. A core contains all writable mapped memory, so a service holding a few gigabytes of heap produces a multi-gigabyte core — the Hook’s 6 GB is ordinary. That has operational weight: dumps can fill a disk and take seconds to write while the process is dead, so production setups cap them, route them to dedicated storage, and sometimes strip unneeded mappings. You want the dump, but you plan for its size.
A containerized service crashes with SIGSEGV but no core file ever appears, even after the developer added 'ulimit -c unlimited' to the entrypoint. What is the most likely remaining cause?
▸Why this works
Why does the core_pattern handler live on the host even for a containerized crash? Because dumping core is a kernel action, and core_pattern is a single host-global kernel setting — there is no per-container core_pattern. When a process in a container segfaults, the host kernel evaluates the host’s core_pattern and runs the host’s handler. So a perfectly correct ulimit -c unlimited inside the container still routes the dump through whatever the node’s core_pattern says, which on a Kubernetes node is often a path or handler your container never sees. The lesson: configure core capture at the node level, not just the image.
Symbols, build-id, and the dev-vs-prod mismatch
You have the dump — why does gdb print nothing but ??? The answer is almost always a build mismatch, and it trips experienced engineers more than beginners because it looks like debugger failure when it is actually a process-control gap.
A core dump is raw memory and register values — it has no idea what function 0x4a3f10 is. To turn addresses into function names and line numbers you need the matching debug symbols, and “matching” is precise: every ELF binary carries a build-id, a hash baked in at link time, and the dump records the build-ids of the binary and shared libraries that were loaded. gdb (or debuginfod) uses that build-id to fetch the exact .debug file. If you point gdb at a binary that was recompiled — even from the same source with a different timestamp or compiler flag — the build-ids differ, the symbol table no longer lines up with the addresses in the dump, and you get the Hook’s garbage stack: ?? frames and meaningless line numbers. This is the “works in dev, cores in prod” trap: your local rebuild is not the prod binary even if the source is identical.
The professional setup separates concerns: ship a stripped binary to production (small, no .debug), keep the full debug info as a separate .debug file matched by build-id, and let debuginfod or your symbol server serve it on demand. Then a core from prod unwinds against the exact symbols even though the running binary carried none. Verify the match explicitly — readelf -n core and readelf -n binary show the build-ids; if they differ, stop, because every frame gdb prints is a lie.
Unwinding a SIGSEGV after the fact
With a matched core and symbols, the autopsy is mechanical. bt walks the saved stack frames from the crash inward, using DWARF CFI (call-frame information) to find each caller’s frame even with a custom frame layout. The top frame is where the fault hit; info registers shows the faulting address (on x86 the accessed address is in a register the kernel records), and a SIGSEGV at address 0x0 screams null-deref while a wild high address suggests a corrupted pointer or use-after-free. You can frame N into any caller, print locals (subject to the optimization caveats from the previous lesson), and read the _exit code: a SIGABRT core usually means an assertion or a libc-detected heap corruption (free(): invalid pointer), pointing you at the allocator’s own diagnosis rather than the raw fault. The dump is a single frozen instant — no replay, no stepping — so you reconstruct the story from the frozen stack, the registers, and the heap, which is exactly why a use-after-free that only sometimes crashes becomes findable: the one captured instant is enough. Now when you see ?? frames in gdb, your first question is not “is the dump corrupt?” but “do my build-ids agree?” — run readelf -n on both before reading a single frame.
gdb opens a 6 GB prod core but the backtrace is all '??' frames with no symbol names, while the same gdb shows a clean stack on a core from your laptop. Most likely cause?
- 01What three gates decide whether a crashing containerized process produces a usable core dump?
- 02Why does a prod core sometimes unwind into ?? frames, and how does build-id fix it?
A core dump is the kernel’s snapshot of a dying process — its address space, every thread’s stack, and the registers — written so you can perform the autopsy after the process is gone, turning an unreproducible crash into a frozen instant you can examine at leisure. But capture is gated. ulimit -c must be non-zero, and it defaults to zero in many container images and systemd units, so the very first crash often produces nothing. /proc/sys/kernel/core_pattern decides destination: a path template or a pipe to a handler like systemd-coredump, retrieved with coredumpctl. Critically, core_pattern is a single host-global kernel setting with no per-container override, so a container crash is routed by the node’s pattern through the node’s handler — which is why a correct ulimit inside the container still yields nothing if capture is not configured at the node. Plan for size, too: a core holds all writable memory, so multi-gigabyte dumps like the Hook’s 6 GB are ordinary and can fill disks. Reading a core needs symbols matched by build-id, the link-time hash recorded for the binary and every loaded library. Aim gdb at a recompiled binary — even from identical source — and the build-ids diverge, the symbol table no longer aligns with the dump’s addresses, and bt unwinds into ?? garbage. That is the works-in-dev, cores-in-prod trap: your local rebuild is not the prod binary. The professional path ships stripped binaries, keeps full .debug matched by build-id, serves it via debuginfod, and verifies the ids with readelf -n before trusting a frame. With a matched core, the autopsy is mechanical: bt walks the frames via DWARF CFI, info registers gives the faulting address (0x0 means null-deref, a wild address suggests use-after-free or corruption), frame and print read the caller chain, and a SIGABRT core points at an assertion or libc’s own heap-corruption diagnosis — all from one frozen instant, no replay. Now when you face an unreproducible crash in production, your checklist starts with two questions before touching gdb: is ulimit non-zero on the node, and does the binary in your hand share a build-id with the one that died?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.