The Blast Radius Budget: A 90-Minute Spike on AI Patch Churn

A reviewer opens a pull request written by a coding agent and finds that the failing test now passes, which is the outcome everyone wanted. The diff, however, touches a config loader, a logging helper, and a fixture file that nobody mentioned, and the actual three-line fix is buried under ninety lines of tidying. That mismatch between the stated task and the delivered patch is the thing worth measuring, and it can be measured inside a single afternoon. This spike does not ask whether the agent is good. It asks a narrower, falsifiable question about the shape of its output, and it pre-registers the number that would make a team abandon the approach. Ninety minutes, one hypothesis, one small script, and a written exit line at the end regardless of which way the evidence falls. The hypothesis, fixed before the first patch The hypothesis reads: when a free-tier coding model is asked to make one named test pass, more than half of the changed lines land outside the symbol that the test exercises. Framed that way it can die cleanly, because six patches with a median out-of-symbol share at or below 0.35 would be enough to reject it. Two secondary rules go into the same document: any patch that edits a file absent from the task's allowed list kills that run immediately, and any hunk the meter cannot classify is reported rather than guessed. Pre-registration matters because the alternative is deciding after the fact that ninety lines of reformatting were probably fine. The thresholds are arbitrary, which is acceptable as long as they are chosen before the data arrives and are not quietly relaxed afterwards. Fifteen of the ninety minutes go to writing this down, and no code is written until it is signed off. Fifteen minutes: building the meter Git already carries the information the meter needs. With zero context lines, each hunk header ends with the enclosing function or class context that git detected, so a diff hunk can be classified without parsing source in the target language. #!/usr/bin/env bash # blast.sh — out-of-symbol share for a single patch # usage: ./blast.sh patch.diff parse_config set -euo pipefail patch_file="${1:?patch file required}" target="${2:?target symbol required}" awk -v target="$target" ' /^@@/ { ctx = $0 sub(/^@@[^@]*@@/, "", ctx) # keep only the trailing context text gsub(/^[ \t]+/, "", ctx) if (ctx == "") { noctx++; inside = 0 } else { inside = (index(ctx, target) > 0) } next } /^[+-]/ && !/^(\+\+\+|---)/ { if (inside) in_sym++; else out_sym++ } END { total = in_sym + out_sym if (total == 0) { print "0 0 0 0.000 0"; exit } printf "%d %d %d %.3f %d\n", in_sym, out_sym, total, out_sym/total, noctx+0 } ' "$patch_file" The output columns are in-symbol lines, out-of-symbol lines, total, ratio, and the count of headers with no context. That last column is the honesty valve, because a file whose hunks never carry context will report every line as outside the target and produce a spectacular false kill. A ratio without the noctx count is a number pretending to be a measurement. The scope check sits beside the meter and uses plain set arithmetic on file names, which is more reliable than reading a diff and trying not to blink. git diff --name-only > /tmp/touched.txt printf '%s\n' src/config/loader.py tests/test_config.py > /tmp/allowed.txt comm -23 0.60 on at least four of six runs kill the workflow, record the failure mode any OFFSCOPE file kill that run regardless of ratio any noctx > 0 recompute by hand before concluding The table exists so that the last five minutes of the spike are mechanical. A team that argues about the verdict after seeing the numbers has not run a spike; it has run a meeting with a stopwatch. Where free access changes the arithmetic A six-run sample is the smallest sample that can show a median without collapsing into a single anecdote, and six patches means six model conversations, six red-state confirmations, and six diff extractions. Free model access on MonkeyCode removes the approval step that otherwise turns a six-run sample into a one-run sample, and the free server option lets the harness run somewhere other than the laptop that is already compiling something else. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Neither convenience changes the verdict rule, which is the entire point of writing the rule first. Free capacity is also not a promise of stability, so a run that fails for infrastructure reasons should be logged as a gap rather than counted as evidence in either direction. What the meter cannot see A tiny patch that makes the test pass by special-casing the fixture scores beautifully and fixes nothing, so the meter has to be paired with a behavioural check that the same patch still fails a mutated version of the test. Generated files, vendored code, and languages without a usable function context will inflate the out-of-symbol count and should be excluded from the allowed list in advance. Six runs support a median and nothing else: no confidence interval, no claim about other models, and no claim that today's free capacity will hold next month. Teams that already enforce a per-patch review budget with a working tool should not adopt this, and neither should monorepos where a symbol name appears in a dozen unrelated modules. The spike is aimed squarely at the gap between the test is green and the change is reviewable, which is exactly where agent patches tend to hide their cost. The exit line At minute ninety the spike writes one sentence: either the hypothesis survived and the workflow is killed, or it died and the workflow survives with a narrower prompt and a file allowlist. Everything else, including the meter, is scaffolding that can be thrown away without regret. If six runs sound better than one for your own repository, a free tier and a free server make the first afternoon cheap enough to actually try.