GC basics: what the runtime taxes you for

A service’s p99 latency spikes to 800 ms every few seconds. CPU looks fine. No slow queries. Opening the GC log reveals pauses of 600–700 ms every 4 seconds — the service allocates 1 GB/sec and the runtime is stopping the world to clean up.

What a garbage collector does

A garbage collector trades CPU for memory safety: it walks the heap, marks what is reachable from roots (CPU registers, thread stacks, global variables), and reclaims the rest. You do not call free(); the runtime does it for you. The cost is that the runtime must spend CPU time on bookkeeping — and it sometimes pauses your application to do so.

Three knobs are always in tension:

• Pause time — how long the application stops while GC works.
• Throughput — how much CPU the collector takes away from your code.
• Heap size — how much overhead memory the collector needs to work efficiently.

You cannot minimise all three at once.

The kitchen metaphor

Think of your service as a restaurant kitchen. Cooks (your code) produce dirty dishes (allocations). A dishwasher (GC) cleans them. If cooks generate dishes faster than the dishwasher can clean, dishes pile up — the kitchen must stop and clear the sink. A modern concurrent dishwasher cleans while cooks work; an old stop-the-world dishwasher makes everyone wait. Even the best dishwasher costs water and electricity (CPU). The fast kitchen does not come from a magical dishwasher — it comes from cooks reusing plates.

Why allocation rate matters more than heap size

Total heap size says how much memory the program holds at any moment. Allocation rate says how often new memory is requested. GC cycles are triggered proportionally to how fast garbage accumulates — which is the allocation rate, not the heap size.

A service that holds 4 GB but allocates slowly may barely see GC pauses. A service that holds 100 MB but allocates 1 GB/s churns GC constantly. The lever for tail latency is allocation rate, not heap size.

The mark-sweep cycle

Every tracing GC follows the same skeleton:

1. Root scan — briefly pause, identify roots (registers, stacks, globals).
2. Mark — walk the object reference graph from roots; mark every reachable object.
3. Sweep — reclaim memory for every object not marked.
4. Optionally compact — move live objects together to defragment.
5. Update references — fix pointers to moved objects.
6. Resume — application runs at full speed until the next cycle is needed.

The naive version pauses the application for all of steps 1–5. A 32 GB heap might take seconds — unusable for any latency-sensitive service. Modern collectors reduce or eliminate most STW phases.

Concurrent vs stop-the-world GC

A stop-the-world (STW) collector pauses all application threads while it works. Simple to implement; pauses grow with heap size.

A concurrent collector runs most of its work alongside your application threads, so user-visible pauses are short (sub-millisecond) instead of long (tens to hundreds of milliseconds). It needs write barriers — small code snippets that run on every reference write to keep the collector informed. The barrier costs ~2–10% CPU; the payoff is short pauses.