Modules and imports deep: sys.modules, finders and loaders, and the circular-import trap

import is a dict lookup — after the first time

By the end of this section you’ll know exactly why moving two lines of code turned 14 seconds into 1.3 — and how to diagnose the same pattern in your own codebase.

import json does something expensive at most once per process. The first time, Python locates the module, executes its entire top-level code in a fresh namespace, and stores the resulting module object in sys.modules — a plain dict keyed by module name. Every later import json, anywhere in the process, is a dict hit: nanoseconds, no I/O, no re-execution. Two consequences run the daily show. First, module top-level code is run-once process init — that is precisely why the Hook’s import pandas cost 12 seconds in every worker: each fresh process pays the first-import price for the whole transitive chain. Second, modules are process-global singletons: settings.DEBUG = True mutates the one shared object every importer sees, which is the entire mechanism behind monkeypatching — and behind the test that mutated a module attribute and quietly changed behavior in an unrelated suite. Deleting an entry from sys.modules forces re-execution on the next import; that is what tools like pytest plugin reloaders exploit, and what you should almost never do by hand.

The machinery: finders, loaders, and the sys.path anatomy

On a cache miss, Python walks sys.meta_path — a list of finders. Stock CPython installs three: one for built-ins, one for frozen modules, and PathFinder, which walks sys.path directory by directory. The finder that claims the name returns a ModuleSpec carrying a loader; the import system then creates an empty module object, registers it in sys.modules before executing anything, and asks the loader to run the module code into that namespace. Hold on to that ordering — registered first, executed second — it is the single fact that decides which circular imports survive.

sys.path itself has a loaded gun at index zero: when you run python script.py, entry zero is the script directory (with -m or the REPL it is the current directory). Local files outrank the standard library. The classic: a helper named random.py in the working directory, and three layers down some library does import random and gets your file. The error arrives far from the cause — AttributeError: module 'random' has no attribute 'choice' deep inside a third-party stack trace; if your random.py itself imports random, you get the circular-import variant instead. Python 3.13 finally prints a hint suggesting you rename the shadowing file; on anything older, the diagnostic move is import random; print(random.__file__). Discipline that prevents the class: absolute imports by default, relative imports (from . import db) only for intra-package references — and never name a module after anything in the standard library.

Packages: init.py executes, and the namespace-package caveat

import pkg.sub is two imports: Python imports pkg first — which means pkg/__init__.py executes — then pkg.sub. That makes a heavy __init__.py a tax on the entire package: the popular convenience pattern of re-exporting everything (from .client import Client, ten more lines) forces every consumer who wanted one small submodule to pay for all of them. The 12-second Hook chain was exactly this shape — an __init__.py aggregating submodules that aggregated pandas. Keep package inits to re-exports of cheap names, and push heavy wiring into the modules that need it.

Drop the __init__.py and you get a namespace package: importable, no init code, and — the actual feature — multiple directories on sys.path can contribute portions that merge into one logical package (company.auth from one distribution, company.billing from another). The honest costs: no package-level initialization hook at all, a slightly slower scan (every path entry must be consulted), and the accident mode — forgetting __init__.py silently produces a namespace package instead of an error, which masks layout mistakes until packaging time. If you are not deliberately splitting one namespace across distributions, keep explicit __init__.py files.

Circular imports: why from-import is the one that dies

When you encounter a ImportError: cannot import name 'X' from partially initialized module, the instinct is to shuffle import order — but the real question is which kind of import your code uses and when it demands the name. Understanding that distinction unlocks the right fix instead of the lucky one.

Module a imports b; halfway through executing b, it imports a back. Because a was registered in sys.modules before its body started executing, the cycle does not recurse — b simply receives the half-initialized module object: whatever a had bound above its import b line exists; everything below does not yet.

# a.py                          # b.py
config = {"region": "eu"}       import a            # gets half-built a: fine
import b                        # from a import helper  ← ImportError!
def helper(): ...               def use():
                                    return a.helper()   # call time: a is complete

import a in b survives because it only binds the module object and defers attribute access to call time — by then a has finished executing. from a import helper demands the attribute right now, mid-execution, and raises ImportError: cannot import name 'helper' from partially initialized module 'a' (most likely due to a circular import) — CPython literally names the disease. The fixes, ranked: restructure — extract the shared piece into a third module both import, killing the cycle (best, and usually reveals a real layering flaw); late import — move the import inside the function that needs it, deferring resolution past init (pragmatic, hides the dependency); import the module, not the name — import a plus a.helper() at call time (cheapest, leaves the cycle in place).

Together these three options cover every real circular-import situation: without the restructure option you patch the symptom and the architectural flaw stays; without knowing the late-import option you might be tempted to delete real behaviour trying to make the cycle go away.

Import-time side effects, main, and the -m flag

Everything above compounds into one rule: module level is for definitions, not actions. A module-level db = psycopg.connect(...) runs during import — in every process that transitively touches the module, including the test runner that just wanted a constant, at a moment when config may not be loaded and event loops do not exist. The lifecycle home for that work is an explicit startup hook — the lifespan pattern from the web-services unit — not import time. The smell test: importing any module of yours should be free of I/O, and python -X importtime -c "import yourpkg" is the profiler that proves it.

Last piece: how a file becomes a program. python pkg/tool.py runs the file as __main__ with no package context — __package__ is empty, so from . import db raises ImportError: attempted relative import with no known parent package, and sys.path[0] becomes pkg/, re-arming the shadowing gun. python -m pkg.tool instead imports pkg properly (running its __init__.py), executes tool as __main__ with package context, and puts the current directory — not the script dir — at sys.path[0]. Scripts that live inside packages need -m; that is also why console-script entry points (next lesson) generate shims that import your module instead of executing the file.