Test isolation and flakiness: making tests deterministic

Why tests flake: shared state and uncontrolled non-determinism

If a test passes on your machine and fails on CI, you are not looking at a mystery — you are looking at one of two precisely-named root causes, and knowing them lets you fix the problem in minutes rather than hours of git-bisect.

A deterministic test gives the same verdict every run, in any order, on any machine. Flakiness is the absence of that property, and it has exactly two roots. The first is shared mutable state — two tests touch the same thing and one leaves it dirty. The classic offenders: a module-level singleton (a cache, a connection pool, a config object loaded once at import), a shared database row both tests read and write, or a global clock everyone reads. The second is uncontrolled non-determinism: Date.now(), Math.random(), freshly generated UUIDs, the host timezone and locale, and timing — sleeping a fixed number of milliseconds and hoping the work finished.

Shared state produces order-dependence: test B passes only because test A ran first and happened to seed the singleton. Reorder them — or run B alone — and B fails. This is why a suite that is green when run whole can be red when you run one file with .only.

// shared-state leak: a module-level counter both tests mutate
import { test, expect } from "vitest";
import { idCounter } from "../src/ids.js"; // module-level singleton: let n = 0

test("A: first id is 1", () => {
  expect(idCounter.next()).toBe(1); // passes only if it ran first
});
test("B: ids are sequential", () => {
  const a = idCounter.next();        // 2 if A ran, 1 if B ran alone
  expect(idCounter.next()).toBe(a + 1);
});
// Run B with --testNamePattern="B" and "A: first id is 1" silently breaks
// the *next* time someone runs the whole suite — the failure has moved.

Runners fight this with isolation. The unit of isolation in Vitest’s default pool and node:test is the file: each test file gets a fresh worker (a separate process or worker thread), so module-level state is born clean per file. That is process isolation — hermetic, because a leaked singleton in file X cannot reach file Y. But within one file, tests share the module graph and the same global object: that is shared-context isolation, and it is where leaks live. Knowing which boundary you are inside tells you whether beforeEach is enough or whether the leak crosses files.

Determinism: control time, randomness, and the environment

You make a test deterministic by removing every input the test does not explicitly own. Pin the clock with fake timers (vi.useFakeTimers() / vi.setSystemTime(...)) or, better for production code, inject a clock so the unit under test calls clock.now() and the test supplies a frozen one — dependency injection beats global patching because it has no teardown to forget. Seed randomness the same way: inject an id generator or a seeded PRNG rather than reaching for Math.random() or crypto.randomUUID() inside business logic. Freeze the timezone by running CI with TZ=UTC — a single env var that eliminates an entire class of date flakes, because date math that crosses a DST boundary behaves differently in America/New_York than in UTC. And never sleep on wall time to wait for async work — await the actual promise or advance fake timers; a setTimeout(50) that works on your laptop loses the race on a loaded CI box.

// flaky: reads the real clock and a real random source
test("token expires in one hour", () => {
  const t = issueToken();
  expect(t.expiresAt).toBe(Date.now() + 3_600_000); // both sides drift
});

// deterministic: freeze time, inject the id
import { vi, test, expect, afterEach } from "vitest";
afterEach(() => vi.useRealTimers());

test("token expires one hour after issue", () => {
  vi.useFakeTimers();
  vi.setSystemTime(new Date("2026-06-05T00:00:00Z"));
  const t = issueToken({ now: () => Date.now(), id: () => "fixed-id" });
  expect(t.expiresAt).toBe(Date.parse("2026-06-05T01:00:00Z")); // exact, always
});

The tradeoff sits between the two isolation modes. Full process isolation per file is hermetic but slow: spawning a worker costs roughly tens of milliseconds, so a thousand-file suite pays seconds to tens of seconds in pure process startup before a single assertion runs. Shared context (a single thread, --no-isolate / pool: 'threads' with isolation off) is fast but leaks state between tests in the same file. The senior balance is to isolate state cheaply, not expensively: reset the singleton in beforeEach, roll back the database in afterEach, and reserve costly process isolation for the few suites that genuinely mutate global module state and cannot be cleaned up in place.

Parallel database isolation: rollback, schema-per-worker, or containers

Tests that hit a real database in parallel are the hardest isolation problem, because the database is shared mutable state by definition. There are three strategies, trading speed for fidelity. A transaction per test, rolled back at the end: each test opens a transaction, does its writes, and the fixture rolls back in teardown so nothing persists — this is the fastest, costing roughly 1–3 ms of DB setup per test, and tests never see each other’s rows. Its limit: you cannot test anything that depends on a real commit — COMMIT-triggered triggers, deferred constraints, multi-connection visibility, or code that opens its own transaction. A schema or database per worker: each parallel worker gets its own namespace and truncates tables between tests; truncate-all runs in tens of milliseconds, slower than rollback but able to test real commits, with workers fully isolated from each other. Ephemeral containers (testcontainers): spin up a fresh database per worker or per suite — maximum fidelity (exact engine version, real extensions), but startup is measured in seconds, so you run far fewer of them and amortize across many tests.

// transaction-per-test fixture (Postgres + a pool): fast, auto-clean, no leaks
import { test as base } from "vitest";
import { pool } from "../src/db.js";

export const test = base.extend({
  db: async ({}, use) => {
    const client = await pool.connect();
    await client.query("BEGIN");
    await use(client);          // the test runs inside this open transaction
    await client.query("ROLLBACK"); // every write vanishes — next test sees a clean DB
    client.release();
  },
});

// usage: each test is hermetic without truncating or restarting anything
test("inserts a user", async ({ db }) => {
  await db.query("INSERT INTO users(email) VALUES('a@x.io')");
  const { rows } = await db.query("SELECT count(*) FROM users");
  expect(rows[0].count).toBe("1"); // 1, regardless of parallel tests
});

The failure mode: retries that mask a real flake

Here is how a flaky test kills a deploy. A test passes 80% of the time — a real 1-in-5 race, say an un-awaited promise whose write lands after the assertion, or a leaked timer keeping the event loop alive into the next test. Someone, tired of red CI, sets retries: 3. Now the test is green: the chance all four attempts fail is 0.2⁴ ≈ 0.16%, so CI is “fixed”. It is not — the 4% effective failure rate (one run in ~625 still red) is now invisible, and worse, the underlying race is invisible too. For months the suite is green. Then the race condition that the flake was a symptom of — the un-awaited write, the order dependency — manifests in production on a deploy, at the worst possible time, with no test ever having pointed at it. Retries did not fix the flake; they hid the bug the flake was reporting.

The async-leak variant is sharper: a test resolves and reports green before its background work finishes — a fire-and-forget void doStuff(), a timer it never cleared. That work then runs during the next test, mutating shared state and corrupting an unrelated assertion. The failure shows up in the wrong test, which is why these are maddening to chase. Detection is mechanical, not heroic: run with --detectOpenHandles (Jest) or watch for node --test open-handle / “did not finish” warnings, which point at the timer or socket still alive at exit; run tests in randomized order (--sequence.shuffle / --randomize) so order-dependence surfaces as a failure instead of a lucky pass; and treat a flake as a bug — quarantine it (mark it skipped/known-flaky, off the critical path) and fix the root cause, rather than papering over it with a retry budget that launders a real failure rate into a green check.