LLM evals: the regression test for non-deterministic features

A provider bumps your model from one snapshot to the next overnight — same model name, same prompt, same temperature. Nothing in your repo changed, so CI is green and you deploy on Friday. By Monday support is flooded: the assistant now answers in markdown tables the parser chokes on, and refuses a class of valid questions it used to handle. There was no exception, no failed test, no log line that said “worse.” The only signal was customers, three days late. The thing that would have caught it on Friday — a suite that re-ran your real prompts and scored the output against known-good answers — was never written, because “it worked when I tried it.”

Why “I tried it and it worked” is not testing

A normal unit test asserts f(x) === y. An LLM does not give you that contract: the same prompt at the same temperature can return different text on each call, and even at temperature: 0 you are not guaranteed identical output across model versions or even across requests, because providers batch and route non-deterministically. So the assertion you actually want is not “output equals this exact string” but “output is good enough on the dimensions I care about” — correct, on-format, on-policy, grounded in the retrieved context.

That shift breaks every habit from deterministic testing. You can’t pin one expected value, so you score a distribution of behavior over many examples. You can’t trust a single green run, because the next run may differ. And the failure mode is silent: nothing throws. A model swap, a prompt edit, a new system message, a changed retrieval index — any of these can quietly drop your answer quality by 15% with zero error signal. Evals are the only mechanism that turns “quietly worse” into a number that fails CI.

The golden dataset: real traffic, not happy paths

An eval is only as honest as its dataset. The golden set is a curated collection of inputs paired with known-good outputs (or with a rubric, when there’s no single right answer). The senior mistake is to seed it with the examples you used while building — the easy, clean, on-distribution ones. Those are exactly the cases that already work. A useful golden set is built from production: real user queries, deliberately oversampling edge cases, past failures, and high-value flows. The discipline that pays for itself: every production failure becomes a golden case the same day, so a fix is provable and the same bug can never regress silently again.

Size is a judgment call, not a fixed number. Teams often start useful at 50–200 well-chosen cases and grow into the thousands as categories accumulate. Coverage beats raw count: 80 cases that span your real query categories catch more than 2,000 near-duplicate easy ones. The number that actually matters is not dataset size — it’s how much of your live input distribution the set represents, which is the failure mode we return to below.

Exact checks vs LLM-as-judge

Score each case with the cheapest method that works. Two families:

Prefer programmatic checks wherever the output has any structure — they’re free, instant, and never lie. Reach for a judge only for the genuinely subjective dimensions. A judge means you make one extra model call per case to grade it: hand the judge the input, the output, and a rubric, and have it return a parseable verdict (a yes/no or a multiple-choice score), exactly as OpenAI’s model-graded eval template prescribes.

The judge is itself a noisy instrument — calibrate it

Here is the trap that burns teams: an LLM judge is not a ground truth, it is another stochastic model with documented biases. A well-built judge can reach roughly 80% agreement with human raters — about the level two humans agree with each other — but only when calibrated; an uncalibrated one drifts well below that. The known failure modes are specific and reproducible:

• Position bias. In pairwise comparison, swapping which answer comes first can flip the verdict. Studies of GPT-4 as judge show its preference reversing when answer positions are swapped; the judge model itself is the largest driver of this bias, more than task difficulty or output length. Mitigate by running both orders and averaging.
• Self-preference. A model tends to rate its own generations higher, with a measured linear correlation between how well it recognizes its own text and how much it favors it. Don’t grade GPT-4 output with a GPT-4 judge if you can avoid it.
• Verbosity bias. Judges systematically prefer longer answers, independent of correctness.

The non-negotiable mechanism: validate the judge against human labels before you trust it. OpenAI’s own guidance is to add a meta-eval with human-provided labels to check the model-graded eval. Hand-label a sample, measure the judge’s agreement against those labels, and only ship the judge once it clears your bar (teams target ~75–90% agreement). A judge you never measured is a number generator you’ve mistaken for a test.

Offline gates vs online eval

Two places evals run, and you need both:

• Offline — the eval suite runs in CI against the golden set on every prompt edit, model bump, or retrieval change. It produces a score; a regression gate fails the build if the score drops below baseline (or below the previous release). This is what would have blocked the Friday deploy. Add new production failures to the golden set so the gate widens over time.
• Online — you sample a slice of real production traffic (say 1–5%), score it with the same automated methods, and watch the trend. Online catches what offline structurally cannot: a silent provider model update, a shift in what users are actually asking, retrieval index drift. A/B tests compare a candidate against the live version on real users and tie quality to business metrics.

The failure that survives a green suite

The worst outcome is not a failing eval — it’s an eval suite that is green while real users hit failures. It happens two ways, and both are about trusting a number you shouldn’t.

The first: the golden set doesn’t cover the live distribution. Your suite passes at 95% because it’s full of the queries you anticipated, while production traffic has drifted toward a category you never sampled — a new language, a new question type, a new document format. The eval is measuring a world that no longer matches the one users live in. The defense is online monitoring: track the embedding distribution of real inputs and alert when production queries land meaningfully far from anything in the golden set, then pull those into it.

The second: you trusted a noisy judge. The suite is green because a biased judge keeps scoring a verbose, confidently-wrong answer as “good.” You shipped a number-generator’s opinion, not a measurement. The defense is the calibration step above — and treating a sudden jump in judge scores after a prompt change as a reason to re-check the judge, not to celebrate.