A Pairing Hour on Free-Model Review: Three Dead Ends and the Two-Run Citation Filter

A 214-line backfill migration sat in review for two days. The junior engineer on the pairing rotation had already tried the obvious thing: paste the file into a free model and ask for a review. The reply was a full rewrite of the file, formatted as a diff, with no mention of the one line that mattered — a new NOT NULL column read before its backfill ran. That was the first dead end of the hour. Three more followed before the session settled on something worth keeping: a review packet small enough to re-read, a rubric made of binary questions, and a two-run citation filter that discards every finding the model cannot point at. A note on evidence before any code. Every script below is runnable as written. The review outputs quoted are illustrative shapes, not benchmark numbers, and no model comparison is claimed. The workflow is the argument; the tooling is the artifact. The first question was not about the model The senior on the pairing rotation did not ask which model was being used. The question was narrower: what does the model actually get to see? That reframing did most of the work. A free model asked to review a whole file produces a whole-file opinion, and whole-file opinions are unreviewable — they mix style, structure, and correctness into one blob. A migration needs a different question. It needs someone to ask whether a read can happen before a write. So the packet became the design. Only hunks that can touch money or schema enter it. Everything else stays home. Dead end 1: paste the file The first attempt sent migrations/0042_backfill_plan.sql plus the service that reads it. The model returned a reformatted rewrite with helpful comments and a suggestion to add indexes. The nullable-column bug survived intact. Worse, the rewrite buried the real diff under cosmetic changes, which is exactly the failure mode that makes AI review feel productive while changing nothing. Lesson: full-file context invites full-file editing. Review needs a bounded surface. Dead end 2: the persona prompt The second attempt added a role: "You are a staff engineer performing a production readiness review." The tone got more confident. The findings did not get better. Persona prompts change register, not evidence. The model still had no way to say this read is unsafe unless the packet made the read and the write visible next to each other. Dead end 3: ask for everything The third attempt asked for all issues. Forty-one findings came back. Three were real. The other thirty-eight were naming preferences, speculative index advice, and one confident claim about a function that does not exist in the repository. That ratio is not a model failure — it is a prompt failure. An unbounded request returns an unbounded list, and an unbounded list gets skimmed. Alert fatigue is the real vulnerability in AI-assisted review. The thing we kept: packet, rubric, two runs, one filter Step 1 — Build a risk-scoped packet #!/usr/bin/env python3 # packet.py — emit only the hunks that can break money or schema import fnmatch import subprocess import sys RISK_GLOBS = ["migrations/*.sql", "src/billing/*.py", "src/ledger/*.py"] MAX_LINES = 400 def diff_text(base="origin/main"): return subprocess.run( ["git", "diff", "--unified=3", "--no-color", f"{base}...HEAD"], capture_output=True, text=True, check=True, ).stdout def by_file(text): current, out = None, {} for line in text.splitlines(): if line.startswith("diff --git "): current = line.split(" b/")[-1] out[current] = [] if current is not None: out[current].append(line) return out def main(): kept = [] for path, lines in by_file(diff_text()).items(): if any(fnmatch.fnmatch(path, g) for g in RISK_GLOBS): kept.append(f"### FILE: {path}") kept.extend(lines) body = "\n".join(kept) lines = len(body.splitlines()) sys.stdout.write(f"PACKET_LINES: {lines} | TRUNCATED: {lines > MAX_LINES}\n{body}\n") if __name__ == "__main__": main() Run it, save the output, and read it yourself before sending it anywhere. If the packet is truncated, split it by file rather than trimming hunks — a half hunk is a lie. python packet.py > packet.txt head -1 packet.txt Step 2 — Write a rubric of yes/no questions Five rules, each answerable without judgement, each requiring a quoted line: R1 A column is written only after its backfill completes. Quote both lines. R2 A read path cannot observe a half-migrated row. Quote the guard. R3 A retry or rerun is idempotent. Quote the conflict key. R4 A numeric change has a unit and a rounding rule. Quote both. R5 Anything the packet does not contain is out of scope. Say "out of scope". R5 is the one that stops the invented function. It gives the model a legitimate way to decline. Step 3 — Run the same packet twice PACKET=$(cat packet.txt) for i in 1 2; do curl -sS "$REVIEW_ENDPOINT/review" \ -H 'content-type: application/json' \ -d "$(jq -n --arg p "$PACKET" '{packet: $p, rubric: "R1..R5", format: "json"}')" \ > "run$i.json" done The endpoint is deliberately dumb: it forwards the packet, returns JSON findings, and stores nothing. A free server is enough for that job, which is why the setup below stays cheap. Step 4 — Keep only cited, repeatable findings #!/usr/bin/env python3 # filter.py — a finding survives only if it is quoted and stable across two runs import json packet = open("packet.txt").read() packet_lines = [l.strip() for l in packet.splitlines() if l.strip()] def load(path): with open(path) as fh: return json.load(fh).get("findings", []) def is_cited(finding): quote = (finding.get("quote") or "").strip() if len(quote) < 12: return False return any(quote in line for line in packet_lines) def key(finding): return (finding.get("rule"), (finding.get("quote") or "").strip()[:40]) first, second = load("run1.json"), load("run2.json") seen = {key(f) for f in second} survivors = [f for f in first if key(f) in seen and is_cited(f)] print(json.dumps(survivors, indent=2)) print(f"kept {len(survivors)} of {len(first)}", flush=True) Three filters, each with a cost. is_cited kills hallucinated code. The two-run intersection kills phrasing noise. The count line kills optimism — when a run keeps zero findings, that is information, not a failure to tune the prompt. Where the free tier and the free server fit Disclosure: This article was prepared as part of MonkeyCode's product outreach. The two-run requirement doubles the request count, which is precisely why a free model allowance matters here. The operator advertises free model access and a free server option; at the time of writing that includes a token allowance described as 10 million tokens. Treat both as availability claims that can change, and re-check the current terms before you build a budget or a CI job around them. The free server is a reasonable host for the thin /review endpoint because the service holds no state. Two caveats worth writing down before anyone depends on it: cold starts make the first request slower than the rest, and a free tier is not a place to put a packet containing customer data. Keep the endpoint stateless, keep the packet small, and keep logs off by default. Decision table Situation Signal Action Packet under 400 lines, all five rules answerable Both runs keep ≥1 cited finding Route to human review with the quotes attached Packet truncated TRUNCATED: True in header Split by file; never send a partial hunk Runs disagree completely Intersection is empty Treat as unknown, not as clean; review manually Findings cite lines absent from packet is_cited drops most The packet is too small or the rubric is too vague Risky surface not expressible as a glob RISK_GLOBS misses files Stop; this workflow does not fit the repo Limitations and who should skip this The filter has a real false-negative mode: a genuine bug the model flags only once gets discarded. Two runs reduce noise and also reduce recall, and that trade is only acceptable when a human still reads the packet. Skip this if the code cannot leave your network, if the risky surface resists file-glob scoping, or if your process needs a single authoritative sign-off rather than a triage aid. The method is a prefilter for a human, not a replacement for review. If you want to try the two-run loop without standing up infrastructure, the free access and free server option are the shortest path to a working endpoint — start with one migration PR and see whether the filter keeps anything worth reading.