Property-based testing with Hypothesis: invariants, shrinking, and the example database

Properties over examples

When you reach for a property-based test instead of another hand-picked example, you are making a bet: that you can state what the function guarantees more precisely than you can enumerate the inputs that matter. That bet pays off more often than most engineers expect.

An example test pins behavior at a point: f(19.99) == expected. A property pins behavior over a space: for all x in some domain, an invariant holds. The craft is in knowing the reusable invariant shapes, because almost every good property is one of four. Roundtrip: parse(dump(x)) == x — the workhorse for serializers, codecs, ORMs. Idempotence: normalize(normalize(x)) == normalize(x) — for canonicalizers and cleanup passes. Commutativity / order-insensitivity: merge(a, b) == merge(b, a) — for CRDT-ish merges, set arithmetic. Oracle: the optimized implementation agrees with a slow, obviously-correct reference — float path versus Decimal path in the Hook, the new parser versus the old one during a rewrite. Writing the property forces a sharper question than writing examples does: not “what does this return for 19.99” but “what does this function actually guarantee” — and functions whose guarantees cannot be stated are usually the ones hiding bugs.

@given and strategies

from hypothesis import given, strategies as st

@given(st.lists(st.tuples(st.text(), st.integers())))
def test_roundtrip(rows):
    assert parse(dump(rows)) == rows        # the invariant, over the whole space

@given(st.builds(Order,
                 id=st.integers(min_value=1),
                 qty=st.integers(min_value=0, max_value=10**6)))
def test_total_non_negative(order):
    assert order.total() >= 0

@given turns a test into a property: Hypothesis calls it repeatedly (100 times by default) with inputs drawn from the strategies. st.integers, st.text, st.lists, st.floats cover primitives with constraints; st.builds constructs objects by calling a callable with generated keyword arguments; @st.composite lets you write custom strategies with internal invariants (a sorted pair, a syntactically valid identifier). The strategies are deliberately adversarial: st.text() produces empty strings and exotic Unicode, st.floats() with no bounds produces nan and infinities, lists come empty and enormous. That hostility is the point — the generator has no intuition about which inputs are “reasonable”, which is exactly the intuition that kept 2.675 out of three years of example tests.

Shrinking and the example database

When an example fails, Hypothesis does not stop — it starts working. The shrinker repeatedly mutates the failing input toward “simpler” — smaller integers, shorter lists and strings, fewer structural elements — re-running the test each time and keeping any candidate that still fails, until it reaches a local minimum. That is why the report says Falsifying example: rows=[('', 0)] instead of dumping a 400-element list of Unicode soup: the minimal case is the difference between a bug report and an archaeology project, and it is the single feature that makes property failures readable. The minimized failure is then written to the example database — a .hypothesis/ directory next to your tests — and on every subsequent run Hypothesis replays known failures first, before generating anything new. Locally this means a red property stays deterministically red until fixed: no luck required to reproduce. The honest CI caveat: the database is local and normally gitignored, so a failure found on CI is not in your local database. For that, Hypothesis prints a @reproduce_failure(...) decorator with the failure (under print_blob=True settings); paste it onto the test locally and you replay the exact CI input. Promote genuinely important cases to permanent regression tests with @example(...), which runs them unconditionally forever.

Budgets: max_examples, deadline, and CI reality

Each property runs max_examples times per invocation — default 100. The budget math is worth doing once: 80 properties × 100 examples × 3 ms of pure logic ≈ 24 seconds, invisible. One property that touches a database at 50 ms per example costs 5 seconds by itself — properties belong on pure logic, and the few that legitimately need I/O deserve a lowered max_examples. Profiles institutionalize this: settings.register_profile("ci", max_examples=1000) for the nightly grind, a 25-example “dev” profile for the inner loop, selected with settings.load_profile(...). The second knob bites harder: deadline, default 200 ms per example, raises DeadlineExceeded when one draw runs long. On a loaded CI runner, GC pauses and noisy neighbors blow that budget on code with no bug in it — the classic “property test is flaky on CI” report. Check the falsifying example first (deadline does catch real accidental-quadratic blowups); if the input is ordinary, raise the deadline or set deadline=None for that test rather than globally.

Stateful testing, in one honest paragraph

RuleBasedStateMachine generalizes properties from functions to sequences: you declare rules (operations with preconditions) and invariants; Hypothesis composes random operation sequences, checks invariants after every step, and shrinks the whole sequence on failure. This is the tool that finds “put, expire, concurrent delete, then get returns a ghost entry” in caches, schedulers, and connection pools — bugs no single-call property can express. It is also the most expensive technique in this lesson: you maintain a model (a simplified reference implementation) alongside the real system, failures take longer to read, and runs are slower. Reach for it when a stateful core has real invariant structure; do not make it your default.

Where property tests pay — and where they do not

They pay where invariants are crisp and inputs are structured: parsers and serializers (roundtrip), codecs and compression (roundtrip), money and unit arithmetic (oracle against Decimal), normalizers and sanitizers (idempotence), date/timezone math, anything with an inverse function. They do not pay on thin CRUD glue: the only property available for “handler calls repository and returns the row” is restating the mock, and generated inputs add runtime without adding falsification power. The working heuristic: if you cannot state an invariant simpler than the implementation, write example tests and move on — a property that re-derives the implementation verifies nothing. Together, these two categories mean that property tests concentrate value at the mathematical core — encoding, arithmetic, parsing — while example tests carry the CRUD surface; mixing them up in either direction wastes time without catching more bugs.