Contract testing and taming flaky tests

Why a shared integration env is the wrong tool

By the end of this lesson you’ll know how to catch cross-service API breaks before deploy — without spinning up a shared environment — and why “just re-run it” is the symptom of a failing test strategy, not a fix.

When two services deploy on the same release train, an end-to-end test that spins up both is honest: it tests exactly what ships. The moment they deploy independently — the normal state of any microservice estate — that honesty evaporates. A full integration environment must run every service at once, which makes it slow to provision, expensive to keep current, and a flake magnet: any one of N services being mid-deploy fails the suite for reasons unrelated to your change. Worse, it tests the wrong question. You don’t care whether all services are mutually compatible in some staging snapshot; you care whether the version you’re about to deploy won’t break anyone who depends on it, and won’t be broken by anyone it depends on.

Consumer-driven contract testing answers exactly that question without ever running two services in the same process. The insight: the only part of a provider’s API that matters is the part its consumers actually use. So let each consumer record, as a byproduct of its own unit tests, the concrete request/response pairs it relies on — that recording is the contract. Each request/response pair is a single interaction: “when I GET /orders/42, I expect a 200 with a body containing amount_cents as a number.” The consumer’s tests run against a mock provider that replays those responses, so they stay fast and isolated. Then the provider fetches every consumer’s contract and, in its own pipeline, verifies it can satisfy each interaction against the real handler. The checkout rename above would have failed checkout’s provider verification the instant it ran orders’ contract — in checkout’s own CI, before merge, with no orders service deployed anywhere.

The broker and can-i-deploy

The piece that turns two pipelines into a coordinated gate is the broker (the Pact Broker, or its hosted form PactFlow). The consumer publishes its contract there, tagged with its version and the branch/environment it’s heading to. The provider’s pipeline pulls the relevant contracts from the broker, verifies them, and publishes the verification result back. The broker thus holds a matrix: for every consumer version and provider version, has this pair been verified to be compatible?

That matrix is what powers can-i-deploy — the actual deploy gate. Before you release version X of a service to an environment, you ask the broker can-i-deploy --pacticipant orders --version X --to-environment production, and it answers yes only if every service orders will talk to in production has a recorded, passing verification against orders’ contract at version X (and vice versa). It is a query over already-collected results, not a fresh test run, so it is instant. This is the difference from a shared env: instead of hoping a staging snapshot reflects production, the broker knows which exact versions are compatible because each side proved it in its own fast pipeline.

Contract tests are not a replacement for schema or OpenAPI checks — they answer a narrower, sharper question. An OpenAPI diff tells you the spec changed in a breaking way; a contract tells you whether a breaking change actually breaks a real consumer. Renaming a field nobody reads is a breaking schema change but breaks zero contracts, so it deploys freely. Tightening a field three consumers parse passes the schema (still valid JSON) but fails their contracts. Use schema linting as a cheap first filter; use contracts to gate on real coupling.

Flaky tests: the trust-killer

What does a pipeline look like when flakiness has taken hold? Exactly the hook: a nightly suite that fails so often its emails go unread, and two separate teams deploying independently with no shared signal. That state is recoverable — but only if you treat flakiness as a quality problem with an owner, not background noise.

A flaky test passes and fails on the same code with no change between runs. It is more dangerous than a consistently failing test, because a red that means nothing trains engineers to ignore red. Once “just re-run it” is the reflex, your pipeline has stopped being a gate: a real regression hides among the noise, people merge on a hunch, and the whole value of CI — that green means safe to merge — is gone. Treating flakiness as a quality problem in its own right, with a budget and an owner, is the senior move; tolerating it is how a test suite dies.

The causes are a short, recurring list, and naming the cause dictates the fix.

The workflow around these is three moves. Detect: re-run the suite on a schedule and track pass/fail per test over time, so flakiness shows up as a dashboard, not a vibe. Quarantine: when a test flakes, mark it as quarantined so it still runs and reports but does not gate the merge — and file a ticket. This protects the pipeline’s signal while you fix the test, without the worse alternative of deleting it. Fix: drive it to determinism using the table above; quarantine is a holding pen, not a graveyard, and a quarantine list that only grows is itself a failure.

Together, detect → quarantine → fix form the only loop that actually shrinks the flake count; skipping “fix” means the quarantine list grows until you’ve quietly dropped coverage on half the codebase.

Bounded retries are the most misused tool here. Auto-retrying a failed test until it passes does not fix flakiness — it hides it, and it will hide real intermittent bugs (a genuine race in production code) just as readily. The only defensible use is a small, bounded retry (e.g., 2 attempts) on end-to-end tests only, where some environmental flakiness is irreducible, and only when every retry is logged and counted so the flake rate stays visible. Never retry a unit or integration test: those should be deterministic, and a retry there is a confession you skipped the fix.