Concurrency for I/O-bound scripts: threads, async, processes

The GIL: one thread runs Python at a time

Before reaching for any concurrency tool, you need one mental model — because picking the wrong tool can cost you an hour of refactoring for zero speedup, exactly like the engineer in the hook. Every confusing thing about Python concurrency traces back to one rule. CPython has a Global Interpreter Lock — the GIL — and it permits exactly one thread to execute Python bytecode at any instant. Spin up ten threads doing pure Python math and they don’t run on ten cores; they take turns on one, time-sliced. That is why threads do not speed up CPU-bound Python.

The twist that makes threads useful anyway: the GIL is released whenever a thread blocks on I/O — a socket read, a disk read, time.sleep, a database round trip, most C-extension calls. While one thread sits parked waiting for bytes, the GIL is free and another thread runs. So for I/O-heavy work — which is most scripting: HTTP, files, databases — threads genuinely overlap the waiting, and the waiting is where all the time goes.

# This is the whole mental model:
#   CPU-bound Python  -> GIL HELD     -> threads just time-slice one core (no speedup)
#   blocking I/O      -> GIL RELEASED -> threads overlap their waits     (big speedup)
#
# So the only question that picks your tool is:
#   "Is this script WAITING on I/O, or BURNING CPU?"

(Python 3.13 ships an experimental free-threaded build that can disable the GIL, but it is opt-in and not yet the default — assume the GIL holds.)

Threads: the easy I/O win

For an I/O-bound job, concurrent.futures.ThreadPoolExecutor is the lowest-effort win in the language. You submit work; the pool runs it across a handful of threads; each one releases the GIL the moment it blocks on the network, so the waits stack up in parallel.

import concurrent.futures
import requests

def fetch(url: str) -> int:
    return len(requests.get(url, timeout=10).content)

def fetch_all(urls: list[str]) -> list[int]:
    # 100 URLs overlap their network waits across ~16 threads.
    # The GIL releases during each requests.get(), so they truly overlap.
    with concurrent.futures.ThreadPoolExecutor(max_workers=16) as pool:
        return list(pool.map(fetch, urls))

The catch is overhead per thread: each carries a real OS thread with its own stack and a context-switch cost. A pool of tens to low hundreds is fine; tens of thousands of threads is not — you’ll drown in memory and scheduler churn. When you need that many in-flight connections, you’ve outgrown threads and want asyncio.

Asyncio: thousands of connections on one thread

asyncio is single-threaded cooperative concurrency. One event loop interleaves thousands of in-flight I/O operations, and each await is a point where a coroutine voluntarily yields the loop while it waits, so the loop runs someone else. No OS thread per task, far less memory, so it scales to thousands of concurrent connections where threads would choke.

import asyncio
import httpx  # an ASYNC http client — requests is sync and would block the loop

async def fetch(client: httpx.AsyncClient, url: str) -> int:
    resp = await client.get(url, timeout=10)  # yields the loop while waiting
    return len(resp.content)

async def fetch_all(urls: list[str]) -> list[int]:
    async with httpx.AsyncClient() as client:
        # gather schedules all coroutines on one loop; thousands overlap cheaply
        return await asyncio.gather(*(fetch(client, u) for u in urls))

The cost is that asyncio is all-or-nothing. Cooperative means one coroutine that doesn’t yield starves every other. A blocking sync call (requests.get, a plain time.sleep, a heavy json.loads over a huge payload) inside a coroutine stalls the entire loop — every other connection freezes until it returns. So either you go async all the way down (async HTTP client, async DB driver), or you push the blocking/CPU work off the loop with loop.run_in_executor(...) so the loop stays responsive.

Processes: true parallelism for CPU-bound work

When the bottleneck is the CPU — resizing images, parsing big files, hashing, number-crunching — no amount of threading helps, because the GIL serializes the compute. You need real parallelism, which in CPython means real OS processes via ProcessPoolExecutor (or multiprocessing). Each worker is a separate interpreter with its own GIL, so they run on separate cores simultaneously.

import concurrent.futures
from PIL import Image

def resize(path: str) -> str:
    img = Image.open(path)            # CPU-bound work that HOLDS the GIL
    img.thumbnail((256, 256))
    out = path.replace(".jpg", "_thumb.jpg")
    img.save(out)
    return out

def resize_all(paths: list[str]) -> list[str]:
    # ProcessPoolExecutor -> N interpreters on N cores; the GIL no longer serializes.
    with concurrent.futures.ProcessPoolExecutor() as pool:
        return list(pool.map(resize, paths))

Processes aren’t free. Each one pays start-up cost, and arguments and return values cross the process boundary by being pickled (serialized) and shipped over IPC. That overhead only pays off for chunky CPU work; firing a process pool at thousands of tiny tasks, or at I/O work, just buys you the overhead with none of the benefit.

The two classic mistakes

Both real, both common, both the same diagnosis run in reverse. Mistake one: using multiprocessing for an I/O-bound job — the eight-minute fetch script from the hook. You pay heavy process start-up and pickle overhead and eat RAM, for no win over threads, because the bottleneck was never the CPU; it was the network, and processes don’t make sockets return faster. Mistake two: using threads (or asyncio) for a CPU-bound job — resizing 1000 images, or a CSV-crunching script — and seeing zero speedup, then staring at it baffled. The GIL serialized your Python compute the whole time; threads only ever overlap waiting, and there was no waiting to overlap. That CSV script stayed exactly as slow with threads as without, until someone switched it to ProcessPoolExecutor and it finally spread across the cores.

The diagnosis is always the same single question: is the script waiting on I/O, or burning CPU? Waiting → threads or asyncio. Burning → processes. Get that one classification right and the tool is obvious; get it wrong and you add complexity for no speedup.