AI / LLM Integration
LLM evals: multiple-choice review
Crux Multiple-choice synthesis across the evals unit — golden sets, programmatic vs judge scoring, judge calibration, offline gates vs online sampling, and the green-suite failure modes.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each mirrors a call you make shipping a real LLM feature — not a definition to recite, but a tradeoff to weigh when the suite is green and you have to decide whether you actually believe it.
Goal
Confirm you can connect golden-set design, scorer choice, judge calibration, and the offline/online split — and spot when a passing eval is still lying to you.
Quiz
Completed
A provider silently swaps your model snapshot overnight. Your repo is unchanged, CI is green, quality drops 15%. Which layer is structurally capable of catching this, and why?
Heads-up CI runs on code changes. A provider-side snapshot swap changes behavior with no diff in your repo, so a commit-triggered gate never re-evaluates against the new model. Online sampling is the layer built for exactly this.
Heads-up Online evaluation exists precisely for this: sample real traffic, score it with the same automated methods, and the quality trend drops when provider behavior shifts. It is detectable; you just need the online layer.
Heads-up Temperature 0 reduces but does not guarantee determinism, and guarantees nothing across model-version changes — the new snapshot can produce different output at temperature 0. It cannot catch this.
Quiz
Completed
You seed a golden set from the clean examples you used while building the feature. The suite scores 96% but users hit failures. What is the design error?
Heads-up Coverage beats count. 80 cases spanning real query categories catch more than 2,000 near-duplicate easy ones. The fix is representing the live distribution, not inflating the number.
Heads-up The scorer is not the problem here — the dataset is. A stricter scorer on the wrong, easy cases still measures a world users don't live in.
Heads-up There is no evidence of a model change. The suite is green because it tests the easy cases you anticipated, not the production traffic that actually fails — a dataset-coverage gap, not a model regression.
Quiz
Completed
Your feature returns a JSON object with a required `status` enum plus a free-text `rationale`. What is the right scoring strategy?
Heads-up Wasteful and risky — you pay an extra model call and inherit judge bias and flakiness to grade fields that have an exact answer. A schema check on the enum is deterministic and free; never hand a structured assertion to a stochastic grader.
Heads-up JSON key order and whitespace vary, and the free-text rationale legitimately differs every run, so exact-string match flags correct outputs as failures. Use structural JSON-match for the fields and a judge for the prose.
Heads-up That is the 'it worked when I tried it' trap: manual checks don't scale, aren't reproducible, and can't gate CI. Spot-checks feed the golden set; they don't replace it.
Quiz
Completed
An LLM-as-judge returns the identical verdict across ten repeated runs of the same case. A teammate calls it 'calibrated.' Are they right?
Heads-up Reproducibility is just low variance. A confidently biased judge is stable and wrong; its stability reads as trustworthiness while it green-lights regressions humans would catch. Calibration is agreement with human labels, not self-consistency.
Heads-up That makes it worse — same-family judging invites self-preference bias, where a model rates its own generations higher. You want agreement with humans, not alignment with the thing being graded.
Heads-up Determinism isn't the issue; the missing step is validation against human labels. A stable judge is fine once you've measured that its agreement with humans clears your bar.
Quiz
Completed
In pairwise LLM-as-judge comparison, swapping which candidate answer appears first sometimes flips the verdict. What is this, and the mitigation?
Heads-up Temperature 0 reduces sampling noise but does not remove position bias, which is a systematic preference for an ordering. The fix is symmetrizing the comparison (both orders, averaged), not lowering temperature.
Heads-up The flip is a real, reproducible judge bias seen even in correct harnesses; studies of strong models as judges show preference reversing on swap. Auditing your labels is good hygiene but won't remove the bias.
Heads-up If order changes the verdict, the judge is not treating them as tied — it is being swayed by position. Symmetrize and average to get a verdict that reflects quality, not placement.
Quiz
Completed
What is the precise mechanism of a regression gate, and what is the senior discipline that makes it tighten over time rather than rot?
Heads-up LLM output isn't a fixed string even at temperature 0; a string-pin gate would fail on legitimate variation. A gate scores a distribution against a baseline threshold, not byte-equality.
Heads-up A gate that doesn't block the merge isn't a gate — it's a dashboard. The point is to fail the build automatically on the change that caused the regression, before it ships.
Heads-up A frozen set goes stale as traffic drifts and stops covering new failure categories. The discipline is the opposite: feed every production failure back in so the gate widens, while tracking the baseline shift.
Recap
The through-line of the unit is one pipeline: a golden set built from real traffic (coverage over count), scored with the cheapest honest method — programmatic where output has structure, a calibrated judge only for open-ended quality — then gated offline in CI and sampled online for drift. The two ways a green suite still lies are a dataset that no longer matches the live distribution and a judge you never validated against humans. Consistency is not accuracy; position, self-preference, and verbosity bias are real; and the discipline that keeps the gate honest is turning every production failure into a golden case the same day.