Your eval suite passes. Does it actually check anything?

Here's a grader I've shipped. Maybe you have too: assert: - type: contains value: approved It's guarding one question — was the loan approved? — and it passes "the loan was approved". Green. Good. It also passes this: "I did NOT approve this. The 'approved' step was skipped entirely." The substring approved is right there, so the check goes green — on an output that says the opposite of what you required. The suite is passing. It just isn't checking anything. I built evalmut to find exactly that, on purpose, before it ships. Mutation testing, but for the grader Code mutation testing (PIT, Stryker, mutmut) flips a > to a >= in your code and asks whether any test notices. If nothing goes red, that test is decoration. evalmut does the same thing one layer up — to your eval grader. It takes a case your grader passes, injects a known defect into the output, and reruns the grader. If the grader still passes a genuinely-wrong output, that's a hole: a class of regression your eval would let ship green. The hard part isn't flipping a boolean. Here a "mutation" is a semantic change whose ground truth you have to establish — and every operator is mined from a documented real-world failure, not invented. (An invented mutation only tests what its author already imagined a check might miss — which is exactly the blind spot you're hunting.) The one rule that makes it trustworthy The whole tool rests on a single invariant: It never infers a hole from a verdict flip. It infers one only from (output-proven-wrong AND grader-passed) — where "wrong" is established against the case's own ground truth, independently of the grader being tested. So an operator applies to a case only where it can prove the mutant's polarity: provably wrong (a defect) or provably still-correct (an equivalent). Where it can't — no number to corrupt, no answer span to truncate, a field the grader doesn't judge — it returns N/A and stays out of the score. A reported hole is never a guess about an ambiguous mutant. No LLM-as-judge anywhere, so a run reproduces byte-for-byte. I pointed it at its own dependency evalmut grades through gradecore, a deterministic grading engine. So I ran evalmut against gradecore's own graders: $ evalmut run demos/dogfood_gradecore.py mutation score 91.4% (32 caught / 35 applied) holes 3 (1 blind spot, 2 coverage gaps) Three real holes — and the part I'm proudest of is that it was fair about them. It called the one broken check a blind spot (a present check that's broken), and the two is-json scopings coverage gaps (a missing check, not a broken one) — because is-json only ever promised to check that keys are present, never their values. A tool that cries "broken!" at a correctly-scoped check is a tool you learn to ignore. It took eight rounds of adversarial self-critique to get the false-positive rate to zero and keep it there; every false hole those rounds found is now pinned by a regression test. There's a short paper in the repo working through the method and the honesty guarantee. Try it on your own suite pip install -e . # depends on gradecore evalmut run your_suite.py If it comes back 100%, your graders earned it. If it doesn't, you just found the outputs your eval waves through. → github.com/egnaro9/evalmut (Built the usual way I work — agents do a lot of the typing, I read every diff and decide what ships.)