Your agent loop is teaching the model to cheat

Wrapping a loop around an AI coding agent is the obvious next move once single-shot prompting stops working. Run it, score it, retry if the score is low. Two walls show up right after: The score goes up and the work is still wrong. The agent learned to satisfy the grader, not the task. Failure is a dead end. Every piece of the loop exists, but a failed run never feeds the next one. Both are environment design problems rather than model problems, and they don't go away as models get stronger. A stronger model finds the shortcut faster. I want to walk through both using a concrete, readable example: an RFC that recently landed in Ouroboros, an open-source Agent OS. The design is written up in issue #1917, implemented in #1916. You can read the whole thing yourself, which is why I'm using it rather than describing something abstract. Wall 1: you handed the answer key to the candidate Most agent harnesses render the acceptance criteria straight into the worker's prompt, including the command that will grade it and the assertion it will be graded against. The motive is reasonable: if the agent knows how it will be checked, it can aim at the right target. Ouroboros did exactly this. _build_success_contract_block rendered verify_command and Expected output: into the worker's instructions. A second leak was harder to spot: on retry, the failure reason carried the assertion's repr(), which rode result.error back into the next prompt. Once the agent can see the assertion, satisfying the assertion is cheaper than satisfying the requirement. The RFC names this directly: a struggling worker's cheapest path is to game the assertion string rather than implement the acceptance criterion, with a postmortem (seed_2be2907edc07) attached. That is textbook reward hacking. You think you are measuring capability; you are measuring the ability to copy an answer. The fix: hide it unconditionally Both leak paths have to close. Closing the forward one alone accomplishes nothing: Forward: _build_success_contract_block now renders only the AC description and expected_artifacts. The harness verifies independently, so the worker never sees the grading logic. Backward: the verify-gate failure reason no longer embeds the assertion repr(). Retry hints come from a dedicated assertion-safe builder (orchestrator/retry_hints.py) that filters the assertion string out of every fragment, including the 2000-character tail of command output. That 2000-character tail is the part worth copying. Plug the main path, forget the log tail, and the leak is still open. The RFC also records that a "disclosure level" config knob was proposed and rejected. An information barrier that can be turned off will get turned off on some afternoon when someone is behind schedule, and nobody will notice, because the scores will look better afterward. What a stuck agent gets instead Hiding everything would leave the worker flailing, so the RFC pairs it with a hint loop. The next round's instructions are reconstructed from what the session actually did: the tool-call trace, the evidence manifest (reusing deliver_gate.load_ac_evidence_manifest, read-only), and the verifier's outcome. Not from the assertion. Disclose the answer Trace-based hints Agent sees "the assertion requires output == X" "you called A and B, artifact C is missing, verification failed at step 3" Cheapest path fabricate X actually produce C Does a rising score mean rising capability? No Yes This is information asymmetry, the same arrangement human exams use: the examiner knows the answer, the candidate only learns where they went wrong. Wall 2: failure is a dead end From the RFC: every piece already existed (the verify gate, the run-to-eval chain, evolve_step, the Ralph driver, focus.select_evolution_focus), but nothing connected them. A failed run never entered formal evaluation. Failure was terminal and surfaced as BLOCKED. A rejected evaluation never entered evolution. Also terminal. The loop existed as three disconnected segments. Failures were reported, not digested. The fix: chain run to eval to evolve Three constraints carry the weight: 1. Failed runs also chain into evaluation. The _run_succeeded gate is relaxed, so any run that produced a session chains into formal evaluation. Fail-open is preserved: an enqueue failure never flips the run's result. 2. A rejected evaluation triggers a budgeted evolution loop. Nobody reimplemented a convergence loop here. The evaluate job's terminal path enqueues the existing evolution machinery when final_approved is False. The new piece is a Gen1 bridge: the run's seed plus the chained evaluation's multi-AC checklist get projected into lineage events, so evolve_step replays the plain run as Generation 1 and starts Generation 2 already focused. 3. Only failed ACs go to the next generation; passing ones freeze. This one buys the most. Making it work required a checklist-to-ACResult converter that satisfies a strict bar: complete index coverage, verbatim ac_content, and semantic_ac_key identity. That strictness is what lets focus.select_evolution_focus freeze the passing ACs. A loop that does not freeze what already passed will redo work it already got right, burning tokens and breaking correct implementations along the way. The symptom is a score that oscillates between generations. The loop has to be able to stop A loop will not stop on its own. Ouroboros reuses Ralph's existing stop conditions: QA pass, convergence, oscillation detection, grade regression, and wall clock. execution.auto_evolve_max_generations defaults to 3, clamped to 1..10. BLOCKED only happens after the budget is exhausted. Oscillation detection and grade regression are the two that catch false convergence: a score bouncing A to B to A to B, or a generation worse than its parent. Both halt the loop instead of burning more tokens. One more fix. evolution/loop.py had a bare except with three silent paths to evaluation_summary=None. It now records a rejected summary carrying the failure reason, which preserves fail-closed focus semantics while making the failure durable. Swallow an exception in a single run and you are wrong once. Swallow it in a loop and the error is amplified across N generations while your logs show nothing. Cheap gates before expensive ones The same pattern runs through the rest of the design. Evaluation is tiered: Mechanical (free, deterministic checks), then Semantic, then Multi-Model Consensus. Anything rejected at layer one never reaches an LLM judge. The interview stage uses a number instead of a tier. Ambiguity is quantified as the inverse of weighted clarity: Ambiguity = 1 - Sum(clarity_i * weight_i) A seed spec cannot be generated until it is = 0.95. Two mathematical gates, and the README states the idea behind both: don't write until it's clear, don't stop until it's stable. What to check in your own loop Don't show the grading criteria to the candidate. Audit whether your harness leaks assert strings back through error messages, log tails, or retry prompts. Close the forward and backward paths, and don't make it a config option. Failure needs a next step. Failed runs should reach evaluation, rejected evaluations should reach evolution, and passing work should freeze. The loop must halt and must detect false convergence. Oscillation, grade regression, wall clock, generation budget. Full RFC: Q00/ouroboros#1917. Implementation: #1916. Design docs live in-tree under docs/hidden-checklist-convergence/ (requirements, architecture, implementation). The project is github.com/Q00/ouroboros: MIT, local-first, and it sits in front of 13 runtime families. If you're running multi-generation agent loops in production, how do you handle false convergence? That's the part I've seen the fewest good answers to.