Test sharding and flaky-test quarantine

Sharding: turn 40 serial minutes into total/N

In ten minutes you’ll understand why sharding by file count barely helps, how flakiness turns a healthy suite red 95% of the time, and what you actually do about it instead of adding retries.

A suite that runs serially has a wall-clock equal to its total CPU time. Sharding splits the tests across N parallel runners — a CI matrix — so each runner executes only a subset and wall-clock drops to roughly total / N + overhead. The runner gives you a slice flag; CI gives you the matrix that launches N copies of the job, each with a different slice index.

# GitHub Actions: a matrix fans the same job out into N parallel shards
jobs:
  test:
    strategy:
      fail-fast: false           # one shard failing must not cancel the others
      matrix:
        shard: [1, 2, 3, 4, 5, 6, 7, 8]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --shard=${{ matrix.shard }}/8

Each shard runs --shard=i/8, executing roughly one eighth of the tests. Then a final job merges the per-shard reports into one result — you do not have eight separate pass/fail verdicts, you have one. The mechanics are: fan out by slice index, run the subset, collect the partial reports, merge.

The trap is how the runner picks each shard’s subset. The naive split is by file count or alphabetical order — divide 600 files into 8 groups of 75. That looks balanced and isn’t, because tests are not equal in duration: the slow integration tests cluster, one shard inherits most of them, and that shard becomes the straggler. Wall-clock is bounded by the slowest shard, not the average, so if shard 3 takes 18 minutes while the other seven finish in 6, your suite takes 18 minutes and seven runners sat idle. You bought parallelism and got one fast lane plus seven empty ones.

# The fix: balance by TIMING, not file count.
# Record per-test durations from past runs, then partition so each
# shard carries ~equal total time (the slow tests get spread out).
npx playwright test --shard=3/8   # runner reads timing data to balance slices
# Jest: --shard plus a timing-aware sequencer; many CI tools (Knapsack,
# CI's own split) do the same — feed them last run's durations.

Diminishing returns: overhead sets the floor

Sharding is not free per runner. Each shard pays a fixed startup cost — checkout, dependency install, warming the cache — before it runs a single test. Call that overhead c. Wall-clock is total / N + c. As you raise N, total / N shrinks toward zero but c does not move, so past a point you are adding runners that each spend more time starting up than testing. Eight runners on a 40-minute suite with two-minute overhead gives 5 + 2 = 7 minutes; thirty-two runners gives 1.25 + 2 = 3.25 minutes for four times the machines — and the bill scales with N. The rule of thumb: pick N where total / N is still a few multiples of c. Past that, you are paying for overhead, not speed.

Flaky tests: the math that keeps the suite red

A flaky test passes some of the time and fails some of the time on the same commit — say it’s green 99% of runs. One such test is a nuisance. Now scale it. Test outcomes are roughly independent, so the probability the whole suite is green is the product of each test’s pass probability: P(green) = (1 - flake)^N. Plug in a 1% flake rate across 300 tests: 0.99^300 ≈ 0.05. The suite is green about 5% of the time — it is red on 95% of runs for reasons that have nothing to do with the change under review.

That is the scaling killer, and the damage is social before it’s technical. When the suite is red most of the time on a green codebase, developers learn that red means nothing. They stop reading failures, blind-retry until it’s green, or merge on a hunch. And the moment red stops being a signal, a real failure sails through in the noise — nobody looks twice at one more red build. The suite has become a slot machine.

Detect, quarantine, fix — not blind retry

The senior playbook has three steps, in order:

1. Detect automatically. Re-run a failure once on the same commit. A test that fails then passes with no code change is flaky by definition — flag it and record a flake rate per test over time. The retry here is a classifier, not a remedy: its only job is to tell flaky from real.
2. Quarantine. Move known-flaky tests out of the blocking path — run them in a separate non-gating lane that still records results, so they stop failing the merge while you keep watching them. The gate goes back to meaning something immediately.
3. Fix the root cause. A flaky test has a determinism bug — a timing assumption, a shared fixture, an unawaited promise, an ordering dependency. Fix that, prove it’s stable, return the test to the gate.

Together these three steps mean the gate stays trustworthy the whole time: detect gives you a classifier, quarantine gives you immediate relief, and fix closes the loop so the test earns its way back. Without step 2 you’re just watching flakiness accumulate; without step 3 the quarantine lane grows and eventually nobody looks at it.

The anti-pattern is making global auto-retry (retries: 3 for the whole suite) the standing policy. It looks like a fix because the red goes away, but it does three bad things: it hides flakiness instead of surfacing it for repair, it slows every build by re-running failures, and — the one that bites — it masks tests that are flaky because they catch a real intermittent bug. A retry that turns a genuine race’s red into green on the second attempt is throwing away exactly the signal you needed. Retry is fine as a detection mechanism; it is poison as a permanent gate policy.