Make Your Code Review Agent Write Down How the Bug Actually Happens

I wrote it together with an AI agent: I set the structure, the agent drafted, and I checked and rewrote the final text. The review agent that quadrupled my code I asked an agent to implement a feature. It should have been about 1,000 lines. When I looked again, it was over 4,000. The implementation agent wrote it. The review agent made it grow. I run a personal project where development is fully automated. I write an issue in Linear, one agent implements it, and a second agent reviews the result. If the reviewer finds something, the implementer fixes it and the loop runs again. If the reviewer finds nothing, the issue is closed. Right after I added the review agent, the implementation ran away from me. It started fixing edge cases that will never happen Take an issue like "if a task is due today, show it at the top of the list". The review agent came back with findings like these: No defense against an attacker who accesses the DB directly and writes an invalid due date No recovery path if the config file is corrupted by hand No retry if the network drops while fetching due dates If an attacker can write to my database directly, security is already gone. That is not a review problem, and no amount of defensive code in the app will help at that point. But the implementation agent dutifully fixed every one of them. Each finding added a layer of defense, and each new layer produced a new finding. The code kept growing with every round, and I ended up with four times the implementation I had planned. The cause was clear: I had let an agent write the review agent's definition. Ask an agent for a "perfect review agent" and it will give you one. A perfect reviewer looks at everything, including things that do not matter. And a reviewer's thoroughness converts directly into implementation size. So my first move was to narrow the scope, by hand: Only report blocking or major problems that reproduce under normal use. Do not cover unrealistic manual state tampering, inputs that are never generated in practice, or defense in depth that assumes an attacker. That cut the noise a lot. But it is a symptom fix. It is a list of things not to look at, and any kind of concern that is not on the list still gets through. Extending the list forever felt wrong, so the next day I went looking for research on how to build review agents. What I found was a different idea: instead of restricting what the reviewer may talk about, require a specific form of evidence for every claim. The idea came from a technique called semi-formal reasoning. The hint: semi-formal reasoning The technique comes from a March 2026 paper by Ugare and Chandra. arXiv:2603.01896, Shubham Ugare, Satish Chandra, "Agentic Code Reasoning" An agent that reads code and reasons about it without running it will, if left to think freely, guess without evidence. It confuses a same-named function in another file with a standard library one, or misses an edge case. On the other end, full formal verification with Lean or Coq is far too heavy for a real codebase. Semi-formal reasoning sits in between. The reasoning stays in natural language, but it is forced into a certificate-like template. The paper's template has five parts: Function trace table: every function inspected, with file, line, and the confirmed behavior Data flow analysis: how the important variables travel between functions Semantic properties with explicit evidence: each claim backed by a code snippet or condition Alternative hypothesis check: verifying in the code that no specific condition changes the behavior Final conclusion: derived only from the evidence in 1 through 4 The key point is that it does not forbid topics. It demands a form of evidence for every claim, so a claim without evidence cannot be produced. The paper's abstract reports patch equivalence accuracy going from 78% to 88%, and 93% on patches generated by agents. Doing all five parts makes the output long, and I read every review result by hand, so that was too heavy for me. I did not adopt the technique as is. I borrowed the idea only. For every finding, the reviewer must write down how you would operate the app to hit the problem. That is all. In the paper's terms it is closest to part 3, evidence-backed claims. A finding without a failure scenario is not a finding I added one line to the reviewer's prompt: Every finding must include a concrete failure scenario in the form "input or state → execution path → failed result". A candidate that cannot be written this way is not a finding. With this, findings like "this might become a bug" or "I am worried about this" are no longer possible. Here is why it works. Take the earlier "what if an attacker rewrites the DB" finding. The first slot, "input or state", cannot be filled. There is no path under normal use that reaches that state. If the reviewer tries anyway, the scenario starts with "an attacker breaks into the server", and that is not a violation of anything in the issue, so the agent has to withdraw the finding on its own. In other words, making the reviewer write down how you get there forces the agent to decide for itself whether the finding is realistic. No human said "do not look at this". The agent judged it while writing the scenario and dropped it. That is why the ban list does not need to grow. Findings that survive look like this. Same issue as before, "due today goes to the top", after the change: tasks/list.ts:42 [major] Tasks created late at night in Japan time are not treated as "today" Failure scenario: a task due today at 23:30 exists and the list is opened at 22:00 → isToday compares dates in UTC → the task is still due today in Japan time but is treated as tomorrow's, so it does not go to the top That is a situation that can genuinely happen, and a human reading it can decide on the spot whether it needs fixing. The old findings read like "no defense against X". They never said which operation would make X happen, so a human could not tell whether to act on them. Since this change, I have rarely disagreed with a finding from the review agent. Takeaway If I were building a review agent from scratch, this is the first line I would add to the prompt: Every finding must include a concrete failure scenario in the form "input or state → execution path → failed result". A candidate that cannot be written this way is not a finding. Requiring one form of evidence worked far better than growing a list of things not to report. References Shubham Ugare, Satish Chandra, "Agentic Code Reasoning" (arXiv:2603.01896, March 2026) https://arxiv.org/abs/2603.01896 Prasanth Aby Thomas, "Meta shows structured prompts can make LLMs more reliable for code review" (InfoWorld, April 1, 2026). A report on Meta using the same technique for code review: requiring explicit assumptions and execution-path tracing reached 93% accuracy on patch equivalence for agent-generated patches. https://www.infoworld.com/article/4153054/meta-shows-structured-prompts-can-make-llms-more-reliable-for-code-review.html