Breakpoints and watchpoints: how INT3, the four debug registers, and conditional eval actually work
A breakpoint is either a 0xCC byte patched into code or one of four hardware debug registers. Conditional breakpoints evaluate on every hit and can run 1000x slower. Watchpoints trap a memory address. Optimized builds reorder and inline, breaking line stepping.
The bug only happened on customer #4,815,162 of a nightly reconciliation job, and only sometimes. A junior set a breakpoint inside the per-row loop and ran under the debugger; the job that finished in 90 seconds was still going after 40 minutes, because the debugger was trapping into the kernel two million times a second to test a condition the engineer had typed into the breakpoint. A senior changed one thing: instead of a conditional breakpoint on the loop, she set a hardware watchpoint on the single account-balance field, told the debugger to continue, and the process ran at nearly full speed until the exact instruction that wrote a negative balance — then froze with the offending stack live. The condition she could not reproduce by reading code became a single trap on a memory address. The difference between 40 minutes and 3 seconds was knowing that one of these breakpoints is a software trap evaluated in the debugger and the other is silicon watching a bus.
Software breakpoints: the 0xCC byte patch
A plain breakpoint is not magic — the debugger overwrites the first byte of the target instruction with 0xCC, the x86 INT3 opcode, and remembers the original byte. When execution reaches that address the CPU raises a breakpoint trap, the kernel delivers SIGTRAP, and the debugger (which is ptrace-attached) wakes up, restores the original byte, lets you inspect state, then to resume it single-steps over the restored instruction and re-patches the 0xCC. This is why software breakpoints are unlimited in count — they cost one byte of patched memory each — but also why they only work on writable code pages: you cannot patch 0xCC into read-only flash or a page you lack write permission for, which is exactly when you fall back to hardware breakpoints.
The patch model has a sharp consequence: a breakpoint set on an address that the target later overwrites or recompiles (a JIT, self-modifying code, a hot-patched function) silently stops triggering, because your 0xCC got clobbered. Delve and gdb hide this, but the mechanism leaks through the moment you debug a JIT runtime.
Hardware breakpoints and watchpoints: four registers, no patching
x86-64 exposes four debug registers, DR0 through DR3, each holding an address, plus DR7 controlling each one’s mode: break on execute, on write, or on read/write, with a length of 1, 2, 4, or 8 bytes. Because the CPU compares these addresses on the bus in hardware, a hardware breakpoint patches nothing and works on read-only pages — but you get only four, system-wide per thread, full stop. A watchpoint is the same mechanism aimed at data: set DR0 to watch an 8-byte field for writes and the CPU traps the instant any instruction stores to it, no matter which code path did the write. That is the senior’s trick from the Hook — instead of asking “which of a thousand lines corrupts this field?”, you let silicon answer by watching the field itself.
The numbers that decide your strategy: a hardware watchpoint costs roughly zero runtime overhead because the comparison is free in the pipeline, while a software watchpoint — what the debugger falls back to when all four registers are taken — single-steps the entire program and compares the value after every instruction, slowing the target by 100x to 1000x. gdb will switch from hardware to software watchpoints silently if you ask for a fifth; info watchpoints and the message Hardware watchpoint versus Watchpoint is how you tell which you got.
You ask gdb to watch a 5th memory location after already setting four hardware watchpoints. The program, previously fast under the debugger, now crawls. What happened?
▸Why this works
Why does a conditional breakpoint cost so much more than a watchpoint on the same logic? A conditional breakpoint still hits in hardware or via 0xCC every single time the address is reached, then the debugger evaluates your condition expression in its own interpreter and decides whether to stop. In a tight loop that is a full trap-into-kernel, context-switch-to-debugger, evaluate, resume cycle on every iteration — easily 1000x the cost of the loop body. A hardware watchpoint on the data flips the question: the CPU stays in the hot loop at full speed and only traps on the one store you care about. Match the trap to the rare event, not the frequent line.
Why optimized builds break stepping
Set a breakpoint on a line in a -O2 build and you may stop on a line that “already ran”, or step into a line that executes three times for one source statement, or be told a local is <optimized out>. None of this is a debugger bug. The optimizer reorders instructions across source lines, inlines callees so there is no call frame to step into, keeps variables only in registers that get reused, and merges identical tails of branches — so the one-to-one map from source line to instruction range that line stepping assumes no longer exists. DWARF debug info records a best-effort line table, but for an inlined, reordered, register-allocated program that table is lossy by construction. The senior workflow is to reproduce under -O0 -g when you can afford it, and when you cannot (the bug only appears optimized, a common Heisenbug), to drop to instruction stepping and read registers directly instead of trusting line numbers and locals.
Debugging an -O2 build, you set a breakpoint on line 40, but it stops and a local shows <optimized out>, and stepping skips lines. The same code at -O0 steps cleanly. Best read?
- 01Contrast software breakpoints, hardware breakpoints, and watchpoints by mechanism and limits.
- 02Why do conditional breakpoints in hot loops kill performance, and why does -O2 break line stepping?
A breakpoint is one of two physical things. A software breakpoint patches a 0xCC INT3 byte over the first byte of an instruction; on hit the CPU traps, the kernel delivers SIGTRAP to the ptrace-attached debugger, which restores the byte, lets you look, then single-steps and re-patches — unlimited in count but only on writable code, and silently defeated by JITs that overwrite the patched page. A hardware breakpoint uses one of just four debug registers DR0-DR3, gated by DR7 for execute/write/read modes and 1-8 byte lengths, comparing addresses on the bus for free and working on read-only pages. A watchpoint points that same register mechanism at data and traps on any store to an address, turning “which of a thousand lines corrupts this field?” into a single trap — the difference between a 40-minute conditional-breakpoint loop and a 3-second answer. Conditional breakpoints are expensive because they trap and re-evaluate on every hit, up to 1000x a hot loop’s cost, while a hardware watchpoint stays out of the loop and fires only on the rare event; ask for a fifth watch and the debugger degrades to single-stepping software emulation at 100-1000x. Optimized -O2 builds reorder, inline, register-allocate, and merge tails, so the source-line table is lossy by construction — stepping skips, lines run out of order, locals read <optimized out>. Reproduce at -O0 -g when you can, and when the bug only shows optimized, step by instruction and trust registers over line numbers. Now when you see a conditional breakpoint crawling through a tight loop or a watchpoint silently missing stores, you know exactly which mechanism to swap and why the switch costs nothing.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.