open atlas
↑ Back to track
CI/CD pipelines CICD · 06 · 05

Merge queues and keeping main green

Required PR checks test stale main, so two independently green PRs can break main via a logical conflict git never sees. A merge queue speculatively builds main + queued PRs before landing, batches for throughput, and bisects to eject the culprit — main green by construction.

CICD Senior ◷ 18 min
Level
FoundationsJuniorMiddleSenior

The team had done everything right. Branch protection on, required checks green, every PR reviewed. And main still broke roughly three times a day. The pattern was always the same: two PRs, each individually green, merged twenty minutes apart, and the second one turned the build red on a line neither author had touched. One Friday it was a rename — someone shortened computeShippingCost to shippingCost and updated all eleven call sites, green. Forty minutes earlier a colleague had opened a PR that added a new caller of computeShippingCost, also green, branched before the rename existed. Git merged both cleanly — no overlapping lines — and main failed to compile. Every push after that was blocked behind a red build until someone bisected, found the orphaned call, and reverted. The checks weren’t lying. They were answering a question nobody had asked: “is this PR green against main as it was when I tested it?” — not “against main as it will be once everything ahead of me lands.”

Why green PRs still break main

If you’ve ever seen two green PRs break main on a line nobody touched, you’ve hit this exact failure. In ten minutes you’ll know the mechanism, why the obvious fixes don’t work, and what a merge queue actually does about it.

A required check on a pull request does not test main. It tests your PR head merged with main as it was at the moment CI ran — a snapshot. By the time your PR actually merges, other PRs may have landed ahead of it, so the main your code was validated against no longer exists.

Git’s merge only blocks textual conflicts: two changes to overlapping lines. It is completely blind to logical (semantic) conflicts, where two changes don’t share a single line yet are incompatible in meaning:

PR A (green):  rename computeShippingCost -> shippingCost, update its 11 callers
PR B (green):  add a NEW call to computeShippingCost(...) in a file PR A never touched
               (branched before PR A existed)

git merge A then B:  zero overlapping lines  ->  merges clean
result on main:      a call to a function that no longer exists  ->  build red

Each PR was green against the main it was tested on. Neither test run could have caught the break, because the combination — A and B together — is a state that existed nowhere until the merge created it. This is why “just re-run the checks on every PR” doesn’t save you: re-running still tests each PR against a main that is stale the instant another PR lands. The higher your merge rate, the more often “tested against a main that no longer exists” ships a broken main.

The merge queue: test main-as-it-will-be

A merge queue (GitHub merge queue, Bors, Mergify) fixes this by serializing integration. Instead of merging a PR directly, it constructs a speculative branch = current main + this PR (plus everything ahead of it in the queue) and runs the full required checks on that combined result. The PR only lands if the queue build — main exactly as it will be after this batch merges — is green.

# GitHub merge queue: the required check runs on the merge_group event,
# i.e. against the speculative branch the queue builds (main + queued PRs),
# NOT just against the PR head. main is protected; nothing pushes directly.
on:
  merge_group:        # <- the queue's speculative build
  pull_request:       # <- still runs per-PR for fast author feedback
jobs:
  required-checks:
    runs-on: ubuntu-latest
    steps:
      - run: ./ci/affected-build-and-test.sh   # keep this FAST — it gates the queue

Nothing reaches main untested against the exact other changes landing with it. The rename PR and the new-caller PR, queued together, would be built on one speculative branch — and that build fails before either merges, so the culprit is held out instead of poisoning everyone. main stays green not by luck or vigilance but by construction: the only thing that ever lands is a combination that was already proven green as a combination.

Batching and bisection: throughput at high merge volume

Testing one PR at a time against current main is correct but slow: at high volume the queue becomes a single-file line, and each CI run only clears one PR. So queues batch — they speculatively build main + PR1 + PR2 + … + PRn together. If the batch is green, all n PRs merge from a single CI run. That is the throughput win: one expensive build clears the whole batch instead of n builds clearing n PRs.

The risk is that one bad PR in a batch turns the whole batch red. The queue handles that by bisecting: it splits the failing batch to find which PR is responsible, ejects the culprit (kicks it back to its author), and re-runs the remaining PRs as a smaller batch. So a single broken PR removes itself instead of blocking the four good PRs queued around it.

# Batch of 4 speculatively built together; the batch build fails.
# The queue bisects to localize the breakage, ejects the culprit, re-runs the rest.
queue batch: [PR1, PR2, PR3, PR4]   -> speculative build RED
  bisect [PR1, PR2] -> green        # breakage is in the second half
  bisect [PR3, PR4] -> red          # narrowed
  bisect [PR3]      -> green
  bisect [PR4]      -> red          # PR4 is the culprit
eject PR4 (back to its author); re-run [PR1, PR2, PR3] -> green -> merge the 3

The trade is direct: bigger batches = higher throughput but more wasted work when one fails (a red batch of 8 throws away a build of 8 and triggers bisection). Smaller batches re-work less but clear fewer PRs per run. You tune batch size to your failure rate — a flaky or break-prone repo wants smaller batches; a disciplined one with rare failures can run large batches and rarely pay the bisection tax.

Why this works

Why doesn’t “require PR checks to pass before merge” keep main green at high merge volume? Because each PR’s checks run against main as it was when that PR was tested, not main as it will be after the PRs ahead of it merge. Git only blocks textual conflicts — overlapping lines — so a logical conflict (PR A renames X, PR B adds a call to the old X in a file A never touched) has no overlap, merges clean, and breaks main. The combination A-and-B is a state that existed nowhere until the merge created it, so neither PR’s CI run could have tested it. The only thing that actually keeps main green is testing the real post-merge combination before landing it — which is exactly what a merge queue’s speculative build does.

Why the queue needs branch protection and a fast gate

Without these two preconditions the queue’s guarantee evaporates — so when you set one up, ask yourself: is there any path to main that bypasses the queue, and how long does the gate check take?

A merge queue only works if main is protected — no direct pushes — and the queue’s combined check is the required gate. If anyone can push to main directly, they sidestep the speculative build and the whole guarantee evaporates: the queue can only promise “everything that lands was proven green as a combination” if everything lands through the queue.

And the gate must be fast. Queue throughput is bounded by the duration of the gating check: the queue can’t clear PR2 until the build that includes PR1 finishes, so a 40-minute required check caps your merge rate no matter how many PRs are waiting. This is the through-line of this whole unit — affected-only builds, caching, test sharding, right-sized runners all exist so the required check stays short, because here that check is what throttles every merge in the org. A slow gate doesn’t just annoy one author; it serializes the entire team behind itself.

Pick the best fit

main breaks several times a day from PRs that were each green on their own branch — logical conflicts like a rename plus a stale caller. You merge a high volume of PRs. What actually keeps main green without throttling throughput?

Quiz

Two PRs each pass all their required checks on their own branch, merge a few minutes apart, and the second merge turns main red — on a line neither author edited. What happened?

Quiz

How does a merge queue keep main green AND maintain throughput when many PRs are landing per hour?

Recall before you leave
  1. 01
    Why can two PRs that each pass all required checks still break main when they merge, and why doesn't re-running the checks fix it?
  2. 02
    How does a merge queue keep main green, and how do batching and bisection preserve throughput at high merge volume?
Recap

Required PR checks don’t test main — they test the PR head merged with main as it was when CI ran, a snapshot that’s stale the moment another PR lands ahead of it. Git’s merge only blocks textual conflicts (overlapping lines); it can’t see logical/semantic conflicts, where two non-overlapping changes are incompatible in meaning — the classic case being PR A renaming a function and updating its callers while PR B, branched earlier, adds a new call to the old name in a file A never touched. Both are green, git merges them cleanly, and main breaks on a line neither author edited, because the combination of both PRs is a state that existed nowhere until the merge created it; re-running per-PR checks can’t catch it. A merge queue fixes this by serializing integration: it builds a speculative branch of current main plus the queued PRs, runs the full required checks on that combined result, and merges only if main-as-it-will-be is green — so main stays green by construction. To keep throughput high at volume the queue batches N PRs into one speculative run, merging all of them if green; if the batch is red it bisects to localize and eject the culprit, then re-runs the rest, so one bad PR removes itself instead of jamming the queue. Bigger batches mean more throughput but more wasted work on failure, so batch size is tuned to the failure rate. The guarantee holds only if main is protected (everything lands through the queue) and the gating check is fast, because queue throughput is bounded by that check’s duration — which is exactly why affected builds, caching, sharding, and right-sized runners matter. Now when you see main broken by two green PRs, you know the fix is structural — a merge queue — not operational, and the first thing to check is whether main is protected and the gating check is fast enough not to throttle the team.

Practice

Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.

recallapplystretch0 of 5 done

Something unclear?

Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.

shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources2
expand
  1. 01
  2. 02

Trademarks belong to their respective owners. Editorial reference only.