Blocking vs non-blocking I/O: two ways to wait

Time a typical request handler and the surprise is how little of it is your code. It reads a row from Postgres, calls a payment API, writes a log line — and spends 95% of its wall-clock time doing nothing but waiting for those to come back. The whole game of backend concurrency is: what does the program do while it waits? Two answers split the entire field. One parks a thread on every wait. The other refuses to park anyone and asks the kernel to tap it on the shoulder when data is ready.

Waiting is the job

A backend is mostly an I/O machine. Disk reads, database queries, outbound HTTP, socket writes — each is slow relative to the CPU (microseconds to milliseconds, while the CPU runs billions of instructions a second). So the design question is never “how fast is my code” first; it is “how does the runtime spend the wait.” Two I/O models give opposite answers, and the choice shapes how the server scales, how much memory it eats, and how it fails under load.

Blocking I/O: one thread per connection

In the blocking model, a thread calls read() and the operating system suspends that thread until bytes arrive. The thread is parked — consuming its stack and a scheduler slot — doing nothing useful. To serve a second connection concurrently you need a second thread, a third needs a third, and so on: thread-per-connection.

This is simple and easy to reason about — the code reads top to bottom, each line waits for the last — but it scales by adding threads, and threads are not free. Each OS thread reserves roughly 1–2 MB of stack, so 10,000 concurrent connections imply on the order of 10+ GB of memory just for stacks, plus thousands of context switches per second as the scheduler shuffles parked threads. The model trades memory for simplicity.

Non-blocking I/O: one thread, many sockets

In the non-blocking model, a socket is set to non-blocking mode and read() returns immediately — either with data or with “not ready yet.” Instead of parking, the thread registers interest in many sockets with a kernel facility — epoll on Linux, kqueue on BSD/macOS — and asks one question: “which of these thousands of file descriptors are ready right now?” The kernel returns only the ready ones, in roughly O(1) time regardless of how many are being watched. The thread services those, then asks again. That loop is the event loop.

One thread can therefore drive tens of thousands of connections, because it only ever touches sockets that have actual work. The cost is a different shape of code: you cannot read top-to-bottom and “wait” — you register a callback (or await) and the loop calls you back later. Logic that was a straight line becomes a set of continuations.

The C10k framing and the real tradeoff

This split was named by the C10k problem (~1999): how do you serve 10,000 concurrent clients on one box? Thread-per-connection hit a memory and context-switch wall; the event-loop model — Nginx, Node.js, Netty, Redis — was the answer. The honest summary:

• Blocking / thread-per-connection trades memory and context-switch overhead for simplicity. Great when connection counts are modest or work is CPU-heavy; the code stays linear.
• Non-blocking / event loop trades code complexity (callbacks, continuations, no parking) for scalability under many concurrent, mostly-idle connections.

Neither is universally “faster.” For I/O-bound workloads with high concurrency, the event loop wins decisively on memory and connection count. For CPU-bound work, a single event-loop thread is no faster than any other single thread — a limit the next lessons make sharp.