io_uring and observability of batching

The flamegraph from the ingest service made no sense. CPU was pinned at 90%, but no business function showed up hot. The widest band, eating a third of every core, was entry_SYSCALL_64 — the cost of entering and leaving the kernel, repeated millions of times a second. The service wrote each log line with its own write(). It wasn’t slow because of what it did. It was slow because of how often it crossed the boundary.

The syscall is a wall, and you keep paying to cross it

Traditional POSIX I/O is one syscall per operation: read(), write(), recv(), send(). Each one is a controlled trap into ring 0 — the CPU saves user registers, switches the page-table and stack to kernel context, runs the handler, then unwinds the whole thing on the way out. That round trip costs roughly 1-5µs, and that’s before the syscall does any actual work. It’s pure overhead, paid per call.

The arithmetic is brutal at scale. A service handling 100k I/O ops/s spends 100,000 × 1-5µs = 100-500ms of every wall-clock second just transitioning — a tenth to half a core gone before a single byte moves. Push to 1M ops/s and traditional read/write can burn a full millisecond per second on transitions alone, plus the cache pollution from blowing away the L1/L2 working set on every crossing. This is exactly the flamegraph in the hook: the work was cheap, the boundary was not.

The classic fix is to cross less often. Buffer many small writes and flush them as one big writev(); one syscall now carries a thousand records. That’s the whole game of batching at the syscall layer: amortize a fixed per-crossing cost over a variable payload. The next step removes most of the crossings entirely.

io_uring: stop crossing the boundary at all

io_uring (Linux 5.1+) replaces “one syscall per op” with two ring buffers mmap’d into memory shared between user space and the kernel:

• Submission Queue (SQ) — userspace writes operation descriptors (SQEs: read this fd, write this buffer) into a slot, then advances a tail pointer.
• Completion Queue (CQ) — the kernel writes results (CQEs) back into a slot and advances its tail.

In the basic mode you still call io_uring_enter() to tell the kernel “I queued N ops” — but that’s one syscall for the whole batch instead of N. The dramatic mode is IORING_SETUP_SQPOLL: the kernel spawns a thread that continuously polls the SQ tail. Userspace submits work by writing memory and bumping a pointer, and the kernel thread picks it up on its own — zero syscalls per op. (One catch worth knowing: if the SQPOLL thread idles past sq_thread_idle, it sleeps and sets IORING_SQ_NEED_WAKEUP; you then owe one io_uring_enter() to wake it. So zero-syscall holds under sustained load, not on a trickle.)

Beyond removing crossings, io_uring unlocks patterns plain syscalls can’t express:

• Linked operations (IOSQE_IO_LINK) — chain SQEs so the next one runs only after the previous completes, e.g. accept → read → write submitted as one dependent unit.
• Provided/registered buffers — pre-register a buffer pool once; the kernel selects a free buffer per op instead of you registering one each time.
• Fixed files — pre-register fds so the kernel skips the per-syscall descriptor-table lookup.

Adoption is now mainstream, not experimental. PostgreSQL 18 (released Sep 2025) shipped async I/O with three io_method modes — sync, worker (the default), and io_uring — where the io_uring backend cuts syscall overhead on cold-cache sequential and bitmap scans (benchmarks report 2-3x throughput gains in cloud-storage scenarios). Note the default is worker, not io_uring, precisely because of the dependency and security concerns below. On the networking side, io_uring shaves single-digit-to-low-double-digit CPU off TLS-proxy and high-fanout socket workloads (the socket layer is where epoll-based proxies spend 70-80% of cycles outside userspace), which is why low-overhead-proxy teams reach for it.

You rarely call io_uring directly — your runtime batches for you

Most services never touch the raw rings; they lean on a runtime primitive that buffers in userspace and flushes as one crossing. The shapes rhyme across languages:

• Node.js — stream.cork() buffers writes in memory; uncork() (deferred via process.nextTick) flushes them as a single _writev() — but only if the stream implements _writev; corking a stream without it can hurt. Pair with backpressure via the write() return value.
• Go — bufio.Writer coalesces small writes; combine with a time.Ticker to flush on a max-wait, giving the classic size-or-time window.
• Java — BufferedOutputStream accumulates until its buffer fills or you flush().
• Python — asyncio.Queue feeding a consumer that drains in chunks (get until empty or count cap).
• Rust — tokio::sync::mpsc channels with a batching loop (recv_many / drain-and-flush on a tick).

Every one of these is the same contract: a bounded buffer with a max-size trigger, a max-wait trigger, and an explicit flush. And every one of them is a place data can silently pile up or get dropped — which is why you instrument it.

The four metrics that tell you the batcher is healthy

A batcher is a tiny queue with a flush policy, and like any queue it can fill, stall, or overflow without throwing an error. Production-grade observability tracks four per-batch metrics; together they let you tune the window and catch backpressure before it becomes data loss.

The failure mode that hides without these is the quiet drop. Facebook’s Scribe log-delivery system is the canonical war story: a buffered, batching pipeline that — under downstream pressure — must choose between blocking the producer (back up the whole app) or dropping messages. If you only watch throughput, a bursty downstream looks fine right up until the buffer overflows and your tail latency p99 metrics start vanishing from the dashboard because the events that carried them got dropped. The dashboard says “healthy” because the survivors look healthy. The senior reflex: buffer depth and drop count are leading indicators; throughput is a lagging one. Alert on drops > 0 and depth > 80% of cap, and the overflow becomes a page you answer, not an incident you reconstruct.

The senior tradeoff: how aggressively to batch

Bigger batches and longer windows save more syscalls and CPU, but every item now waits longer before it ships — directly inflating tail latency. The whole skill is choosing the window against your SLO, and proving the choice with the metrics above rather than guessing.