Threads vs processes: choosing by workload, races under the GIL, and the fork+threads deadlock

Choosing by workload: where each pool actually wins

When you reach for concurrent.futures, the first question is never “how many workers?” — it is “what does a worker actually do while it waits?” The answer determines which pool saves you and which one quietly makes things worse.

concurrent.futures gives both pools one interface, but the physics differ. ThreadPoolExecutor shares the process memory: submitting a task costs microseconds, passing a 500 MB DataFrame to a worker costs nothing — it is a reference. The previous lesson explained the constraint: threads only overlap where the GIL is released, so the thread pool is for I/O-bound work — HTTP calls, database queries, S3 reads — where workers spend their lives blocked in the kernel. Sizing is forgiving (default is min(32, cpu_count + 4)); for I/O you can run dozens of threads per core. ProcessPoolExecutor buys true CPU parallelism by paying rent in serialization: every argument and every return value crosses the process boundary through pickle. That tax is the most common reason “parallelized” code gets slower. Honest numbers: pickling ~100 MB of plain Python objects takes on the order of a second each way, so shipping a large DataFrame to a worker that does 200 ms of computation is a net loss of seconds. The pattern that works: send small descriptions of work (a file path, an ID range), let the worker load its own data, return small results. And batch — executor.map(fn, items, chunksize=100) amortizes per-task IPC overhead that would otherwise dominate when each item takes microseconds.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import multiprocessing as mp

# I/O-bound: threads. 32 workers, each blocked in the kernel most of the time.
with ThreadPoolExecutor(max_workers=32) as pool:
    pages = list(pool.map(fetch_url, urls))          # overlaps for real

# CPU-bound: processes. Pass paths, not data — the args cross via pickle.
ctx = mp.get_context("spawn")                        # explicit start method
with ProcessPoolExecutor(max_workers=8, mp_context=ctx) as pool:
    results = list(pool.map(render_pdf, paths, chunksize=4))

The GIL does not make your code thread-safe

A dangerous half-truth: “Python threads can’t race because of the GIL.” The GIL guarantees one bytecode runs at a time — but count += 1 is three bytecodes (load, add, store), and the 5 ms switch can land between any two of them. Two threads load the same value, both add one, both store: an increment is lost. In a quick test, two threads doing 1,000,000 unguarded += each typically land visibly short of 2,000,000 on 3.9 and earlier — modern CPython (3.10+) switches only between full bytecode lines, making this specific demo usually pass, which is worse: the race is still there, just rarer and unreproducible in tests. The general killer is check-then-act: if key not in cache: cache[key] = expensive() — two threads pass the check before either writes, and the expensive call runs twice (or worse, half-initialized state becomes visible). Individual dict and list operations are atomic in CPython, but any sequence of operations that must be consistent needs a threading.Lock. The discipline is the same as in any language: identify invariants spanning more than one operation and guard them; the GIL changes the probability of the interleaving, never the possibility.

import threading

cache, lock = {}, threading.Lock()

def get_config(key):
    with lock:                      # guard the whole check-then-act
        if key not in cache:
            cache[key] = load_from_disk(key)   # runs exactly once per key
        return cache[key]

fork vs spawn: how a child process is born

If you have ever shipped a service that worked on a Mac and deadlocked silently in production on Linux, the start method is the first place to look. Understanding the difference takes five minutes; debugging the alternative without it took the Hook’s team three days.

multiprocessing creates workers via a start method, and the choice is a correctness decision. fork (historically the Linux default) clones the parent process at the moment of the call: instant, no re-import, all loaded data available copy-on-write. Its fatal interaction is the Hook’s bug: fork() copies the whole address space including every held lock, but only the calling thread survives into the child. Any lock held by another thread at fork time — logging’s lock, an allocator lock inside a C library, a connection pool’s mutex — is locked forever in the child, and the first touch deadlocks. The rule: fork and threads do not mix, and almost everything spawns threads behind your back (logging handlers, grpc, boto3, OpenBLAS). CPython 3.12 started emitting a DeprecationWarning when fork happens in a multi-threaded process, and the default start method is moving away from bare fork (macOS switched to spawn back in 3.8; Linux moves to forkserver in 3.14). spawn starts a fresh interpreter and imports your module — slow (hundreds of ms per worker), requires everything submitted to be picklable and the entry point guarded with if __name__ == "__main__": (otherwise each child re-executes the pool creation: a process bomb), but immune to inherited-lock deadlocks. forkserver is the middle path: a clean single-threaded server process forks workers on demand. Production guidance: set the start method explicitly — mp.get_context("spawn") — and never rely on the platform default, because that is exactly what changed between the laptop and the container.

Shared state across processes: mostly, don’t

Processes share nothing by default, and the escape hatches are expensive or sharp. multiprocessing.Queue and pool results move data by pickling. Manager() objects proxy every access through a server process — convenient, roughly 1000× slower than a local dict. multiprocessing.shared_memory and Value/Array give real shared buffers but reintroduce every race from the threads section without the GIL’s accidental protection, so they need multiprocessing.Lock. The architecture that survives production: workers are stateless functions over their arguments; shared state lives in something built for it (Redis, the database) or is reduced to a final merge step in the parent. If you find yourself reaching for Manager().dict() on a hot path, the design is telling you it wanted threads — or a different decomposition.