Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals

Example scenario (illustrative, not a reported incident): a team's eval harness has returned PASS on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying — the harness simply stopped being able to see that class of bug. Green means "the cases we wrote still pass". It does not mean "the grader can still detect this failure". Those are different properties, and only the second one decays silently. The technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest. 1. Three ways a grader goes blind Exit-code-only grading. The harness records returncode != 0 as a failure and nothing else. A patch that compiles, runs, and prints a wrong number passes. So does one that emits a new warning nobody promoted to an error. Baseline-normalized output. Warnings and diagnostics are diffed against a stored baseline file. When the baseline is regenerated after a noisy commit, the diff quietly absorbs the next real diagnostic. Assertions the patch can edit. The grader greps test output for a string, and the test file lives in the same tree the model may modify. The patch adjusts the expected string; the grader confirms the adjustment. Each of these is a grader weakness, not a case weakness. Adding golden cases will not fix any of them, which is why pruning and shuffling the case set does not surface the problem either. 2. The artifact: a mutant manifest plus an audit loop The audit needs three inputs: a clean git worktree, a list of small patches with known faults, and a harness entry point that can report per-grader status as machine-readable output. A manifest declares, for each mutant, which graders must reject it. # mutants/manifest.yaml (illustrative) mutants: - id: signed-modulo-clamp diff: mutants/signed_modulo_clamp.diff why: replaces a bounded comparison with a modulo must_fail: [unit_clamp, prop_clamp_bounds] - id: unchecked-alloc-size diff: mutants/unchecked_alloc_size.diff why: drops a size guard before memcpy must_fail: [asan_guard] - id: silent-float-narrowing diff: mutants/silent_float_narrowing.diff why: narrows double to float without a cast warning gate must_fail: [warnings_as_errors] The must_fail list is the contract. A mutant that survives a listed grader is a finding; a mutant that fails extra graders is fine and often informative. 3. Step-by-step: wiring the loop Step 1 — Give the audit an isolated worktree Dirty state is the most common cause of false "invalid diff" verdicts. A dedicated worktree keeps the audit off the developer's working tree. git worktree add ../eval-audit HEAD python tools/mutant_audit.py \ --manifest mutants/manifest.yaml \ --worktree ../eval-audit \ --report audit-$(date -u +%Y%m%d).md Step 2 — Apply, grade, reset The runner below is illustrative and has not been executed as part of this article. It expects python -m evals.grade --json to print {"graders": {"": {"status": "pass"|"fail"}}} on stdout. #!/usr/bin/env python3 """Illustrative grader self-audit. Not executed for this article.""" import json, pathlib, subprocess, sys import yaml ROOT = pathlib.Path(__file__).resolve().parent.parent def sh(cmd, cwd): return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True) def reset(worktree): sh(["git", "checkout", "--", "."], worktree) sh(["git", "clean", "-fdq"], worktree) def apply(worktree, diff_path): diff_path = str(diff_path) if sh(["git", "apply", "--check", diff_path], worktree).returncode != 0: return False return sh(["git", "apply", diff_path], worktree).returncode == 0 def grade(worktree): """Returns (graders_dict, error). Never treats a crash as a grader failure.""" p = sh([sys.executable, "-m", "evals.grade", "--json"], worktree) try: return json.loads(p.stdout)["graders"], None except (json.JSONDecodeError, KeyError): return None, f"no JSON from harness (exit {p.returncode})" def audit(manifest_path, worktree): manifest = yaml.safe_load(pathlib.Path(manifest_path).read_text()) rows = [] for m in manifest["mutants"]: reset(worktree) if not apply(worktree, ROOT / m["diff"]): rows.append({"id": m["id"], "verdict": "invalid-diff", "survived": []}) continue graders, err = grade(worktree) if graders is None: rows.append({"id": m["id"], "verdict": "infra-error", "survived": [], "detail": err}) continue survived = [g for g in m["must_fail"] if graders.get(g, {}).get("status") != "fail"] rows.append({ "id": m["id"], "verdict": "caught" if not survived else "blind", "survived": survived, }) reset(worktree) return rows Three verdicts matter, and they are not interchangeable. caught is a healthy grader. blind is the signal the audit exists for. infra-error means the harness itself broke — it must never be counted as a pass, otherwise a crash in the grader loop reads as perfect detection. Step 4 — Report kill rate per grader, not per run A single global number hides the useful part. Aggregate by grader id so a weakness localizes: from collections import defaultdict def kill_rates(rows): seen, killed = defaultdict(int), defaultdict(int) for r in rows: if r["verdict"] not in ("caught", "blind"): continue for g in r.get("must_fail", []): seen[g] += 1 if g not in r["survived"]: killed[g] += 1 return {g: killed[g] / seen[g] for g in seen} A grader below roughly its own historical rate is worth a look, and a grader at zero is definitionally blind. The thresholds are a team convention; they are not a standard, and picking one without a baseline of stable weeks is guesswork. Step 5 — Triage survivors into three buckets Not every survivor is a bug, and treating them all as one leads teams to abandon the audit. Weak grader. The mutant is a real fault and the grader missed it. Fix the grader or add the missing assertion. This is the case worth real engineering time. Equivalent mutant. The patch compiles and looks wrong but is behaviorally identical on every reachable path. Retire it from the manifest with a note explaining why. Missing case. The grader is sound but no golden case reaches the affected code. This is a case-set gap, so it overlaps with coverage work rather than duplicating it. Record the bucket in the report. A manifest that grows without retiring equivalent mutants becomes noise within a few months. 4. Generating candidate mutants without hand-writing all of them Hand-authored mutants are high quality and slow to produce. A model can draft candidates from the function under test plus the current grader list, and a human then decides which are real faults versus equivalents. The cheap part of that loop is generation, and the expensive part is review — which is exactly where the human should stay. This is also the only place MonkeyCode participates in the workflow described here. Its free model access covers the candidate-generation step, and the free server option is a place to run the nightly audit so it does not consume CI minutes or a developer's laptop. The project's outreach materials describe both options; specific quota figures and server hardware change, so read the current numbers on the official page rather than planning capacity from any blog post, this one included. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A generation prompt that respects the constraint the audit needs — compiles cleanly, changes behavior — reads roughly like this: Given the function below and these existing graders [list], propose 5 patches that (a) compile without new errors, (b) change observable behavior on at least one reachable input, (c) plausibly evade at least one listed grader. For each, state the input class that exposes the fault and which grader should catch it. Two rules keep model-generated mutants honest. Every candidate is compiled before it enters the manifest, since a patch that fails to build is not a mutant. And every candidate is reviewed for equivalence by a human, since a model that believes two behaviors differ is not evidence that they do. 5. Verify the audit itself An audit that always prints caught is worse than no audit. Plant a known failure before trusting the report: replace one grader's body with return True in a scratch branch, run the audit, and confirm that grader's kill rate drops to zero and the run exits non-zero. # in the audit entry point blind = [r["id"] for r in rows if r["verdict"] == "blind"] errors = [r["id"] for r in rows if r["verdict"] == "infra-error"] if blind or errors: print_report(rows, kill_rates(rows)) sys.exit(1) If the planted failure is not reported, the problem is the audit, not the graders. This self-check is the difference between a dashboard and a measurement. 6. Limits, and who should skip this The audit assumes deterministic graders and a harness that exposes per-grader status. Flaky timing or network checks make survivors ambiguous, and the first fix should be determinism, not more mutants. It also assumes someone owns the manifest. A stale mutant set stops finding things while continuing to print green — the same failure mode it was built to detect. Skip this if golden cases barely exist yet, if the harness only reports pass/fail with no per-grader granularity, or if a single engineer owns both the harness and every patch under test. The cost lands on the harness owner, and the benefit only appears when grader coverage and patch volume are both large enough for silent blindness to be plausible. Closing Graders rot quietly. A short weekly run that shoves known-bad patches through the real harness converts that decay into a number someone can act on. Teams already running a patch eval loop can start with three mutants and one grader contract; that is enough to learn whether the report is describing the model's patches or the harness's blind spots.