Catch LLM regressions before your users do — a tiny CI gate for LLM output

You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does. I evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: a file you run on every change, that fails the build when it should. Here's how I do it, with runnable, dependency-free code. Make your eval a file you run Keep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — how to judge it: [ {"id": "q1", "question": "What year was the first moon landing?", "answer": "1969", "match": "contains"}, {"id": "q2", "question": "What is pi to two decimals?", "answer": "3.14", "match": "numeric", "tol": 0.01}, {"id": "q3", "question": "Capital of France?", "answer": "Paris", "match": "contains"} ] Your model's answers are a simple {id: answer} object: { "q1": "The first moon landing was in 1969.", "q2": "Pi is about 3.14159", "q3": "The capital of France is Paris." } Now score them. This is the open-source llm-eval-harness — one file, standard library only: python llm_eval.py gold.json answers.json accuracy: 100% (3/3) The match rule is what makes this trustworthy. contains passes "The first moon landing was in 1969." against gold 1969 — an exact-match-only script would mark that wrong and send you chasing a non-bug. numeric reads the first number in each and compares within a tolerance, so 3.14159 matches 3.14. The scorer normalizes first (lowercase, trim, collapse whitespace, drop trailing punctuation), which kills a whole class of false negatives. And it exits non-zero the moment anything fails, so it drops straight into CI: # .github/workflows/eval.yml - name: LLM eval gate run: python llm_eval.py eval/gold.json eval/answers.json That alone catches "did this change break a known-good answer?" on every PR. Accuracy is the wrong number to watch Here's the trap: a change can raise your average score and still break the three questions your biggest customer depends on. The number that actually matters on a change isn't the score — it's which items that used to pass now fail. So compare two runs and diff them. This is one of the pieces the LLM-Eval Starter Kit adds on top of the free harness: from llm_eval_kit import run, diff_runs, load_gold, load_answers gold = load_gold("eval/gold.json") before = run(gold, load_answers("eval/answers_main.json"), name="main") after = run(gold, load_answers("eval/answers_pr.json"), name="pr") diff = diff_runs(before, after) print(f"accuracy {diff['before_accuracy']:.0%} -> {diff['after_accuracy']:.0%}") if diff["regressed"]: print("REGRESSIONS:", ", ".join(diff["regressions"])) raise SystemExit(1) # fail the build accuracy 100% -> 67% REGRESSIONS: q3, q5 diff_runs gives you the regressed ids (passed before, fail now), the fixed ids, and the score delta. Gate CI on regressed and a prompt change can't quietly ship a regression again — the PR goes red with the exact ids that broke. Open-ended answers: an honest LLM-as-judge String rules run out fast. "Is this summary faithful to the source?" "Did it follow the format?" "Is the tone right?" You can't contains-match those. The stable approach is an explicit rubric graded by a model at temperature 0 — not a vibe check. The kit keeps that honest: the rubric is data (criteria + a numeric scale + a pass threshold), and the judge is any ask(prompt) -> str callable, so you bring your own model and key. There's also a deterministic offline judge so your test suite never needs an API key: from llm_eval_kit import score, StubJudge, make_openai_judge gold = [{ "id": "sum1", "match": "rubric", "question": "Summarize the release note in one sentence.", "context": "v2.3 adds retry-with-backoff to the uploader and fixes a memory leak.", "rubric": { "criteria": [ "Faithful to the source (no invented features)", "One sentence, plain language", ], "scale": [1, 5], "threshold": 4, }, }] answers = {"sum1": "v2.3 adds automatic upload retries and fixes a memory leak."} # Offline + deterministic — perfect for CI/tests, no key: acc, failures = score(gold, answers, judge=StubJudge(must_include=["retry", "memory leak"])) # Real judge — bring your own key (OpenAI / OpenRouter / a local server): judge = make_openai_judge(model="gpt-4o-mini", temperature=0) # reads OPENAI_API_KEY acc, failures = score(gold, answers, judge=judge) One rule I never skip: spot-check the judge against a handful of human labels. An LLM judge is a measuring instrument, and an uncalibrated instrument lies confidently. Grade ~20 answers yourself, compare, and only then trust it at scale. When one attempt isn't the whole story For sampled or agentic systems you often care about "did any of k attempts pass?" rather than a single greedy answer. That's pass@k: from llm_eval_kit import score_pass_at_k rate, details = score_pass_at_k(gold, samples_by_id, k=5) # samples_by_id: {id: [answer, ...]} Same gold set, same match rules — you just feed it multiple samples per id. The takeaway You don't need a platform to stop shipping LLM regressions. You need: Your eval as a versioned JSON file with a match rule per item. A scorer that exits non-zero so CI can gate on it. A regression diff so you watch which items flip, not just the average. For open-ended answers, a rubric + a temperature-0 judge you've spot-checked. The scoring core is open-source and free (MIT): llm-eval-harness. If you want the rubric LLM-as-judge, pass@k, regression diffs, and shareable HTML/Markdown/JSON reports ready to run — still zero dependencies — that's the LLM-Eval Starter Kit, launch price $24 for the first two weeks with code LAUNCH (then $39). There's also a short book on the whole method, Practical LLM Evaluation. How are you catching regressions when a model updates under you? I'd genuinely like to hear it.