Test suites that scale: contracts, flake taxonomy, budgets
Each component tier has a public contract — test that, not internals. Whole-DOM snapshots assert everything and thus nothing. Flakes split into timing, order-dependence, and shared state; retry masks all three. MSW: happy-path defaults, per-test overrides. Unit slice under 10 s.
The suite had grown to 4,200 tests and 38 CI minutes, with a 6% chance any given run went red for no code-related reason. The team voted for the obvious mercy: retry: 2 in the Vitest config. Red runs nearly vanished — 6% became a fraction of a percent — and everyone stopped looking at flaky failures, because the third attempt always saved them. Three months later, checkout shipped a double-submit race: clicking Pay twice during a slow response created two charges. There was a test for it. It had been failing intermittently for six weeks — roughly one run in three, exactly like a flake — and retries had been quietly greening it the whole time. The incident review counted 1,800 duplicate charges, a weekend of refunds, and one sentence that ended up on a slide: “Our test knew, and our CI was configured not to listen.” Retry does not fix flakiness — it reclassifies intermittent signal as noise, and intermittent is precisely how real race conditions present. The fixes that actually scale a suite are unglamorous: contracts per tier, isolation hygiene, a flake taxonomy with owners, and time budgets enforced like any other SLO.
What to test: a contract per tier
“What should we test?” has a precise answer once you name each tier’s public contract — the promise consumers rely on, behind which internals may change freely. Design-system leaves (Button, Select, Modal): the contract is render output for prop states plus interaction behavior — keyboard activation, focus trap, aria attributes; test exhaustively here because twelve teams inherit every bug. Feature components (OrderForm, SearchPanel): the contract is a user flow against an API — fill, submit, see success and failure states; test with RTL plus MSW, a handful of flows per component, not one test per prop. Pages and routing: the contract is composition — the right feature components mount with the right data scope; a few integration tests, because E2E covers the rest at higher fidelity and higher cost. And pure logic (validators, formatters, reducers, price math) does not belong in component tests at all: extract it and test it in plain Node where a thousand cases cost two seconds. Inverting any layer burns budget: prop-level exhaustiveness at the page tier produces thousands of slow, brittle tests; flow-level-only coverage at the leaf tier ships a broken focus trap to twelve teams.
The anti-pattern with a smell you can grep for: tests asserting internals — state values, mock call counts on internal modules, CSS class names. Those assertions pin the implementation, fail on harmless refactors, and pass on real breakage (the class is present; the layout is destroyed). Every assertion should be phrased as something a user or a consuming component could observe.
A 400-line whole-DOM snapshot of OrderForm fails after an intentional copy change. The reviewer scans the wall of diff, presses u to update, approves. What is the systemic read?
Snapshots, honestly — and the flake taxonomy
Before you decide whether to reach for a snapshot, ask yourself: could a teammate reading this diff in six months tell whether the change is correct or just different? That question separates useful snapshots from expensive noise.
The honest rule for snapshots: a snapshot is an assertion whose expected value is generated — so it is only as good as a reviewer’s ability to judge its diff. A ten-line toMatchInlineSnapshot of a formatted error or a computed config sits in the test file, readable and reviewable. A 400-line whole-DOM toMatchSnapshot fails on every markup touch, trains the team to press update, and within a quarter the suite contains hundreds of expected values nobody has ever read. If a diff cannot be judged in under thirty seconds, it should have been three targeted assertions.
Flakes deserve the same precision. Three species, three mechanisms, three fixes — and retry addresses none of them:
- Timing — assertions racing async work: waitFor budgets sized for laptops breaching on 2-vCPU CI runners, spinner asserts racing fast mocks, missing
awaiton user-event. Fix: await outcomes (findBy), size budgets per environment, never sleep. Lesson 02’s territory. - Order-dependence — a test passes alone, fails after a specific neighbor: an MSW override not reset, a leaked fake clock, a module-level cache surviving between files,
localStoragewritten and never cleared. Fix: hygiene inafterEach(resetHandlers, useRealTimers, storage clear) plus fresh per-test factories for anything stateful. Detection is mechanical: run withsequence.shuffleenabled and the species surfaces in a night. - Shared state — parallel workers colliding on one resource: same database row, same port, same temp file, a global
Datemock half-applied. Fix: per-worker namespacing or in-process fakes; this species scales with parallelism, so it appears exactly when you speed the suite up.
Retry converts all three from deterministic debugging targets into background radiation — and, as the Hook’s checkout race showed, real intermittent bugs present identically to flakes, so a retried suite is structurally unable to tell you about races. The grown-up alternative is a quarantine lane: a flagged test runs but does not block merges, an owner and a deadline are attached, and the quarantine list is visible and finite. Quarantine admits the signal is broken; retry pretends it is not.
A spec passes when run alone but fails whenever it runs after settings.test.tsx — which overrides an MSW handler to return 500 and has no afterEach. Classify the flake and name the fix that scales.
MSW organization and the time budget
Handler architecture follows one rule from lesson 02’s seam logic: defaults describe the healthy API; deviations live with the tests that need them. A handlers/ directory with per-domain files (orders.ts, auth.ts) exports the happy-path set that setupServer composes; error responses, empty states, and slow responses are expressed via server.use inside the specific test. The moment a 500 lands in shared defaults, every unrelated test inherits a broken API — one team spent two days on “random” failures that were a colleague’s error-case handler committed into the default set. Shared data factories (buildOrder(overrides)) keep handler payloads schema-shaped in one place, so an API field rename is a one-file change instead of a 60-test archaeology dig.
Budgets make all of this enforceable. Numbers that hold up in practice: a pure-logic test runs in well under a millisecond; a jsdom component test costs 50–300 ms (jsdom DOM construction dominates, roughly 10x a node test); so a healthy split is a pure-logic slice under 10 seconds — fast enough to run on every save — and a component suite in single-digit minutes on CI, parallelized across workers. Vitest runs test files in parallel workers by default; per-file isolation costs roughly a third of total runtime, and turning it off (isolate: false) is a perf lever you may pull only when the hygiene above is airtight, because it converts every module-level leak into a cross-file flake. Past roughly ten minutes, shard across CI machines. Treat the budget like an SLO with an owner: when the suite breaches, profile the slowest hundred tests — the usual suspects are component tests that should be logic tests, real timers nobody faked, and a setup file rendering half the app per spec.
▸Why this works
Why does retry feel so cheap and cost so much? Probability does the laundering. With a 6% suite-level flake rate, retry: 2 shows red roughly 0.02% of the time — flakiness seems solved for one config line. But the same arithmetic applies to genuine intermittent bugs: a race that fails one run in three passes a three-attempt gate about 70% of the time per test execution — so a bug like the Hook’s double-charge merges within a day or two of attempts. Retry is indistinguishable from a filter that deletes exactly the failures whose pattern is “sometimes” — and “sometimes” is the signature of races, leaks, and shared-state collisions, the highest-severity class a suite can catch. Quarantine is the honest version of the same mercy: it also unblocks merges, but it keeps the failure visible, owned, and finite instead of statistically erased.
- 01Give the flake taxonomy with one mechanism and one structural fix per species, and explain why retry is the wrong control for all three.
- 02Describe the per-tier contract model and the budget numbers that keep a suite reviewable and fast.
Scaling a test suite is an architecture problem wearing a tooling costume. The load-bearing decision is naming each tier’s public contract and refusing to test anything else: design-system leaves earn exhaustive prop, keyboard, and aria coverage because twelve teams inherit every defect; feature components earn a handful of user flows against MSW; pages earn composition checks; and pure logic gets extracted into plain node tests where a thousand cases cost two seconds. Assertions phrase what a user or consuming component observes — internals-pinning assertions fail on harmless refactors and pass on real breakage, the worst of both worlds. Snapshots obey a reviewability bound: an assertion whose expected value is generated is only as strong as a reviewer’s ability to judge its diff, so ten-line inline snapshots of formatted errors are real assertions while 400-line DOM dumps train the update reflex that lets regressions ride through. Flakes are not weather; they are three mechanisms. Timing flakes race async work and yield to awaited outcomes and environment-sized budgets. Order-dependence is leaked state — unreset MSW overrides, frozen clocks, module caches — killed by afterEach hygiene and fresh factories, detected by shuffling execution order. Shared-state flakes are parallel workers colliding on real resources, and they arrive precisely when you parallelize. Retry fixes none of these mechanisms; it statistically erases the symptom, and because real races present as intermittent failures too, a retried suite is configured not to hear them — the Hook’s 1,800 duplicate charges were a test that knew, silenced by a third attempt. Quarantine with owners and deadlines is the honest mercy. MSW organization mirrors the seam logic: happy-path defaults composed from per-domain handler files, every deviation expressed by server.use inside the test that needs it, data factories keeping payloads schema-true in one place. And budgets are SLOs: logic slice under ten seconds, component suite in single-digit minutes across workers, shard past ten, profile the slowest hundred when breached. None of it is glamorous; all of it compounds. Now when you see retry: 2 in a Vitest config, you read it as a flake-suppression decision that may also be suppressing a real race — and you ask which species of flake prompted it before accepting the config as given.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.