Iterators and generators: paused frames, lazy pipelines, and the file handle that never closed
for calls __iter__, then __next__ until StopIteration. A generator is a paused frame: send/throw/close drive it, yield from delegates. Lazy pipelines process gigabytes in constant memory — but a paused frame holds its resources, so a generator can hold a file open for hours.
The exporter streamed daily CSV snapshots to S3 through an elegant generator pipeline: read_rows(path) opened the file and yielded parsed rows, downstream stages filtered and batched. Then the platform team capped open file descriptors at 1024, and the service started dying with OSError: Too many open files — but only on days with many small tenants. The leak hunt found nothing that forgot to call close. The culprit was control flow: for tenants with zero matching rows, a downstream stage returned early after the first batch attempt, abandoning the pipeline. The read_rows generator was paused at a yield inside with open(path) — the with block never exited because the frame never resumed. A paused generator frame is a live object holding everything in scope: the file handle, the parsed row, the buffers. Garbage collection would eventually run close(), but CPython only finalizes when references drop, and a registry dict kept each pipeline alive. One thousand abandoned generators, one thousand open descriptors, one page at 4 a.m.
The iterator protocol: what for actually compiles to
for x in xs: desugars to: call iter(xs) — which calls xs.__iter__() — to get an iterator, then call it.__next__() repeatedly until it raises StopIteration, which the loop swallows as normal termination. Two roles, often confused: an iterable has __iter__ and can be looped many times (lists, dicts, your collection classes); an iterator has __next__ and an __iter__ returning itself, carries position state, and is exhausted exactly once. The classic bug this distinction explains: pass an iterator (not a list) to two consumers, and the second sees nothing — iteration consumed it. A second subtlety with teeth: StopIteration is flow control, so a stray StopIteration raised inside a generator’s body would silently end iteration instead of crashing; PEP 479 (default since 3.7) converts it to RuntimeError — if you see that error, somebody used next() without a default inside a generator.
Generators: paused frames with an API
A function with yield does not run when called — it returns a generator object wrapping a suspended stack frame: locals, instruction pointer, the works. next() resumes the frame until the next yield, which suspends it again, value handed to the caller. return (or falling off the end) raises StopIteration with the return value attached. The frame’s persistence is the whole feature — state machines without classes — and the Hook’s whole problem: everything the frame references stays alive while it sleeps.
Beyond next(), the full driving API: gen.send(value) resumes and makes the paused yield expression evaluate to value (coroutine-style protocols are built on this — it is how asyncio’s machinery began); gen.throw(exc) raises the exception at the paused yield, letting the generator catch it for cleanup or recovery; gen.close() throws GeneratorExit there, expecting the generator to exit — a finally or with block around the yield runs at that moment. That is the fix for the Hook: a consumer that abandons a pipeline must call close() (or hold the generator in a with contextlib.closing(...)), which resumes the frame one last time so with open(...) can exit. yield from sub delegates an entire sub-generator: values, send, throw all pass through transparently, and the sub-generator’s return value becomes the expression’s value — manual for x in sub: yield x loses send/throw passthrough and the return value.
import csv
from collections.abc import Iterator
def read_rows(path: str) -> Iterator[dict]:
with open(path, newline="") as f: # held open while frame is paused!
yield from csv.DictReader(f) # delegation: full protocol passthrough
def large_orders(rows: Iterator[dict], min_total: float) -> Iterator[dict]:
return (r for r in rows if float(r["total"]) >= min_total)
pipeline = large_orders(read_rows("orders.csv"), 100.0)
try:
for row in pipeline:
process(row)
finally:
pipeline.close() # resumes the paused frame so `with open` can exitA generator opens a file in a with-block and yields rows from inside it. A consumer reads 10 rows, then returns early and keeps a reference to the generator in a dict. When does the file close?
Pipelines and itertools: the vocabulary of lazy processing
Chained generators give you streaming with constant memory: each stage pulls one item at a time, so a 20 GB CSV flows through filtering, transformation, and batching in a few kilobytes of working set — the alternative, materializing lists between stages, costs the full dataset per stage. The numbers are honest, though: a generator resume is a Python-level frame switch, so per-item overhead is roughly 100–200 ns versus ~30 ns for a C-level list iteration step; for million-item hot loops where everything fits in memory, a list comprehension is often faster. Laziness buys memory and time-to-first-item, not raw throughput.
itertools is the standard vocabulary so you stop hand-writing stateful loops: islice (pagination over any iterator — no slicing needed), chain (concatenate streams), groupby (run-length grouping — requires sorted input, the most common itertools bug), tee (split one iterator into n — but buffers everything the slowest consumer has not read, so two consumers at different speeds silently rebuild the list in memory), batched (3.12+, the batching stage everyone wrote by hand for a decade), plus count/cycle/takewhile/dropwhile. These compose into pipelines that read like the data flow they implement — and they are all C-implemented, so a pipeline of itertools stages largely avoids the per-item Python frame cost.
A pipeline stage does `for x in sub(): yield x` instead of `yield from sub()`. Functionally identical for plain iteration — what did the refactor actually lose?
- 01Explain how an abandoned generator leaks a file handle, why GC does not promptly save you, and the two-sided fix.
- 02Compare next/send/throw/close and state precisely what yield from adds over a manual for-yield loop.
Iteration is two dunders: for calls iter() for an iterator, then __next__() until StopIteration — iterables (with __iter__) restart on every loop, iterators (with __next__ returning themselves from __iter__) carry position and exhaust once, which is why sharing an iterator between consumers starves the second. PEP 479 turned stray StopIteration inside generator bodies into RuntimeError, closing a silent-truncation trap. A generator function returns a suspended frame: next resumes to the next yield, send makes that yield evaluate to a value, throw raises at it, close injects GeneratorExit so finally and with blocks finally run — and yield from delegates all of it, values, send, throw, close, plus the sub-generator’s return value, which a hand-written for-yield loop drops. Pipelines of generators process arbitrarily large streams in constant memory and excellent time-to-first-item, at an honest ~100–200 ns per Python-level resume versus ~30 ns for C-level list steps — itertools (islice, chain, sorted-input groupby, buffer-hungry tee, 3.12’s batched) supplies C-speed stages and a shared vocabulary. The production failure is the paused frame as resource holder: a consumer that abandons a pipeline mid-stream leaves with open(...) unexited, the descriptor alive, and the OOM-killer’s quieter cousin — fd exhaustion — on the pager. Close what you do not exhaust.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.