The test pyramid and required checks
Shape the suite as a pyramid — many fast unit tests, fewer integration, a thin e2e cap — then make only the fast, reliable checks block a merge via branch protection. Invert either and the gate turns slow, flaky, and untrustworthy.
The team had 600 end-to-end tests and 40 unit tests. CI took 38 minutes, failed on roughly one run in three, and nobody trusted a red build — the standing advice was “just re-run it.” Two browser tests had been failing for weeks; instead of fixing them, someone marked the whole e2e job informational so merges could proceed. The gate was now decorative. The suite was an upside-down pyramid, and the cost was paid on every single pull request.
The pyramid: granularity bought with speed and confidence
By the end of this lesson you’ll know why the shape of your test suite determines whether your merge gate is trustworthy — and how to configure branch protection so only the right checks block a merge.
A CI test suite is a portfolio, and the test pyramid (Ham Vocke’s “Practical Test Pyramid”, building on Mike Cohn) tells you how to weight it: lots of small unit tests at the base, some coarse integration tests in the middle, and very few high-level end-to-end tests at the top. The shape encodes a tradeoff that runs in opposite directions as you climb.
At the base, unit tests exercise one function, class, or module in isolation, with its collaborators faked or stubbed. They run in milliseconds, fail with a precise stack trace, and never touch a network or a disk — so you can afford thousands of them and run them on every save. Their weakness is fidelity: a unit test passes against your model of the world, not the real database or the real API, so it can be green while the integration is broken.
In the middle, integration tests check that two real parts talk correctly — your code against a real Postgres in a container, a real HTTP call to a stubbed-but-wire-accurate dependency, an ORM against a real schema. They catch the bugs units miss (wrong SQL, serialization mismatches, migration drift) at the cost of seconds each and a heavier setup.
At the top, end-to-end tests drive the whole system the way a user does — browser, app server, database, the lot. They give the highest confidence that the thing actually works, and they are slow, expensive, and, as Vocke puts it, “notoriously flaky and often fail for unexpected and unforeseeable reasons.” That is exactly why the layer must stay thin.
| Layer | Scope | Speed | Relative count | Run in CI when |
|---|---|---|---|---|
| Unit | One function/module, collaborators faked | ms — thousands in seconds | Many (the wide base) | Every push; fail-fast first |
| Integration | Two real parts (code + DB, API + wire) | ~seconds each | Fewer (the middle) | Every push; after unit passes |
| End-to-end | Whole system through the UI | ~tens of seconds to minutes | Few (the thin cap) | Critical paths only; quarantine flaky |
The frequency gradient is the whole game: confidence rises as you climb, but so do runtime, flakiness, and debugging cost. You buy most of your safety cheaply at the base and reserve the expensive top for the handful of critical user journeys — checkout works, login works — that justify the price. When you find yourself wanting to add another e2e test for an edge case, ask whether a fast unit test at the base could prove the same thing for a fraction of the cost.
▸Why this works
Coverage percentage is a weak proxy for safety. It measures which lines executed during tests, not whether you asserted on the right behavior. A test that calls a function and never checks its output can push coverage to 90% while proving nothing; conversely, a small, sharp test of the one branch that breaks in production is worth more than a hundred lines of incidentally-covered glue. Vocke’s guidance: “Test for observable behaviour instead” — assert that inputs x and y produce result z, not that a specific private method got called. Chase behavior, not the number.
The ice-cream cone: why inverting the pyramid hurts
Flip the pyramid and you get the ice-cream cone: a thin scoop of unit tests under a fat bulge of e2e tests — the shape in the hook. It feels safe (“we test the whole app the way users use it!”) and it is a trap Vocke explicitly warns will be “a nightmare to maintain and takes way too long to run.”
Three costs compound. Slow: end-to-end tests dominate wall-clock time, so feedback that should arrive in a minute arrives in forty — and slow feedback is the thing CI exists to prevent. Flaky: the more of the system a test spans, the more moving parts can hiccup — a slow render, a race, a network blip — so failures stop correlating with real bugs. Once a red build is as likely to be noise as signal, people start re-running on red, and the suite has lost its only job. Hard to debug: a failed unit test points at one function; a failed e2e test means “something, somewhere across five services, went wrong” — you get a screenshot and a timeout, not a stack trace.
The fix is not to delete e2e tests but to push assertions down. Most of what a broad browser test checks — validation rules, formatting, error handling, edge cases — can be proven by a fast unit or integration test. Keep e2e for the irreplaceable thing it alone proves: the pieces are wired together and the critical journey completes.
Gates: making the right checks block a merge
A green suite is worthless if anyone can merge red. The enforcement mechanism on GitHub is branch protection: a rule on main that blocks direct pushes and forces changes through pull requests that satisfy conditions. The CI-relevant levers:
- Required status checks — named checks (your unit job, your integration job) that “must have a successful, skipped, or neutral status before collaborators can make changes to a protected branch.” A check that is not in the required list still runs and reports, but it is informational: it cannot block the merge. This is the dial you turn to decide what is load-bearing.
- Require branches to be up to date before merging — the strict setting. With it on, a PR must be rebased onto the latest base and re-pass CI before merging, which closes the semantic merge conflict gap (two PRs that each pass alone but break when combined). The cost is more re-runs; on a busy repo strict can become a merge-queue bottleneck.
- Required reviews — N approving reviews before merge, optionally dismissing stale approvals when new commits land.
Together these three levers form a safety envelope: required checks enforce correctness, the up-to-date setting closes combination failures, and required reviews add a human signal. Without at least the first lever, any of them alone is hollow — a branch with four approvals but no status checks can still merge code that fails every test.
The design decision is which checks to make blocking. The rule of thumb: gate on checks that are fast and trustworthy; keep slow or flaky checks informational. Your unit and integration jobs are required. A flaky full e2e suite should not be required — gating on it means a non-deterministic test can block every merge in the org, and the “fix” people reach for is to disable the gate, exactly as in the hook.
A 9-minute e2e suite fails ~15% of runs from timing flakiness, blocking unrelated PRs. What do you make of it as a required check?
A service has 95% line coverage and a green CI run, yet a serialization bug ships to production. How is that possible?
On a protected branch, which configuration both blocks merging broken code AND avoids a flaky test blocking every PR?
- 01Why does the ice-cream cone (mostly e2e tests) make a CI suite slow, flaky, and untrustworthy — and what's the fix?
- 02What is branch protection, and how do you decide which checks to make required versus informational?
The test pyramid weights a CI suite by granularity: a wide base of many fast unit tests that isolate one module with fakes, a narrower middle of integration tests that check two real parts (your code against a real database or a wire-accurate dependency), and a thin cap of slow, expensive end-to-end tests that drive the whole system. Confidence rises as you climb, but so do runtime, flakiness, and debugging cost — so you buy most of your safety cheaply at the base and reserve the top for a handful of critical journeys. Inverting it into an ice-cream cone makes the suite slow, flaky, and untrustworthy, and the fix is to push assertions down rather than delete e2e. Remember that coverage percentage measures executed lines, not asserted behavior — test what the system does, not which lines ran, because “all green” is not the same as “actually safe.” Gating is what makes a green suite matter: branch protection with required status checks blocks merging code that fails the fast, reliable jobs, the strict up-to-date setting closes the semantic-conflict gap, and you make only trustworthy checks blocking. A flaky e2e suite belongs in a non-blocking quarantine lane with a small required smoke subset, not in the merge gate — and you keep CI fast with parallelism, sharding, and fail-fast so the gate never becomes the bottleneck. Now when you see a team clicking “re-run” on every red build, you’ll know the fix: reshape the suite toward the base, and gate only on what’s fast and deterministic.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.