A Better Model Improved the Numbers. It Didn't Fix the Product.

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you 25+ CLI commands, an MCP server, 7 export formats, and a bundled git rule pack. The field test report is the source for every number in this article: 4 models, 13 corpus types, 394 trajectories, 844+ deterministic tests passing. The strongest cloud model tested was meta-llama/llama-3.1-8b-instruct with 123 pass / 168 inconclusive / 101 fail — better than every other model, but still not enough to solve the safety problem on its own. Release notes · Changelog There is a comforting fantasy in AI engineering. When a system struggles, you tell yourself the problem is the model. Spend a little more, use a stronger one, and the product gets better. Sometimes that is true. One of the most useful outcomes from the CauterRule field test was learning exactly where that story stops being true. Why we ran cloud models After fixing the parser, prompt, result-reset, and corpus timestamp issues, we finally had a benchmark we could trust. That was the moment cloud comparison became worth doing. We ran two working cloud models through the full corpus: openai/gpt-4o-mini and meta-llama/llama-3.1-8b-instruct. The point was not just to get better numbers. It was to answer a sharper question: Are the remaining problems mostly local-model weakness, or are they product problems? The answer turned out to be both, but mostly the second. All-model comparison: local vs cloud The broadest comparison is the full-corpus view. Model Type Total rows Total candidates Pass Pass % Inconclusive Fail omlx-openai-Llama-3.2-3B-Instruct-4bit local 393 379 72 19.0% 189 118 omlx-openai-Qwen3-4B-Instruct-2507-4bit local 373 373 93 24.9% 209 71 openai/gpt-4o-mini cloud 394 394 77 19.5% 248 69 meta-llama/llama-3.1-8b-instruct cloud 394 392 123 31.4% 168 101 Three things jump out. First, the cloud models removed most of the remaining availability and formatting ambiguity — both produced near-perfect parse reliability. Second, meta-llama/llama-3.1-8b-instruct produced the strongest overall pass count of any model we tested: 123 passes from 392 candidates. Third, gpt-4o-mini was still operationally valuable because it was stable, cheap, and fully runnable across all 394 trajectories. But there is something more subtle in this table that matters more than the ranking. Even the strongest tested model still produced 168 inconclusives and 101 hard fails. That is 269 out of 392 candidates — 69% of the output was not clearly usable. Better models improved the system. They did not settle the product. Curated comparison: where model quality actually mattered most The curated corpora are the cleanest lens for this comparison. Curated corpus Local Llama 3.2B Local Qwen 4B Cloud GPT-4o-mini Cloud Llama 3.1 8B golden 8P / 2F 3P / 4I / 3F 6P / 4I / 0F 7P / 0I / 3F failures/positive 15P / 3I / 12F 15P / 9I / 6F 14P / 13I / 3F 22P / 5I / 3F failures/negative 1P / 2I / 5F 1P / 4I / 5F 0P / 4I / 6F 3P / 1I / 6F successes 1P / 16I / 2F 2P / 8I / 10F 0P / 6I / 14F 3P / 6I / 11F nearmiss 4P / 4I / 6F 5P / 5I / 4F 2P / 4I / 8F 6P / 1I / 7F noisy 1P / 3I / 1F 4P / 1I / 0F 1P / 3I / 1F 5P / 0I / 0F corrections 2P / 2I / 1F 2P / 1I / 2F 3P / 2I / 0F 3P / 1I / 1F If you want the shortest honest reading of that table: cloud models clearly help on quality. meta-llama/llama-3.1-8b-instruct was the strongest model tested. And the positive-case corpora improved much faster than the safety corpora. That last point matters more than the ranking itself. Winner-by-corpus view This view is a better way to summarize what actually improved. Corpus Best model Why it matters golden Local Llama 3.2B by raw pass count, Cloud Llama 3.1 8B by strongest cloud showing regression anchor / specificity failures/positive Cloud Llama 3.1 8B best practical extraction benchmark successes none were convincingly strong safety / over-triggering check failures/negative none were convincingly strong (all 5-6 fails) rejection quality nearmiss Cloud Llama 3.1 8B, but still mixed trigger precision noisy Cloud Llama 3.1 8B strongest clutter-handling corrections Cloud GPT-4o-mini and Cloud Llama 3.1 8B both strong correction-to-rule synthesis The cloud models won where learning-from-failure was easiest. They did not decisively win where restraint mattered most. The raw corpus results confirm the pattern The raw corpora add volume and show the same pattern at scale. Raw corpus Local Llama 3.2B Local Qwen 4B Cloud GPT-4o-mini Cloud Llama 3.1 8B raw/opencode 13P / 2I / 9F 13P / 7I / 5F 12P / 10I / 3F 18P / 3I / 4F raw/synthetic 18P / 101I / 26F 36P / 87I / 22F 28P / 91I / 26F 38P / 81I / 25F raw/ci 8P / 49I / 52F 9P / 90I / 11F 7P / 99I / 4F 11P / 60I / 39F raw/sibling-repos 0P / 9I / 1F 0P / 9I / 1F 0P / 7I / 3F 0P / 10I / 0F raw/corrections 2P / 2I / 1F 3P / 1I / 1F 2P / 3I / 0F 3P / 0I / 1F raw/cross-session 2P / 1I / 2F 1P / 2I / 2F 2P / 2I / 1F 4P / 0I / 1F Cloud Llama 3.1 8B led on raw/opencode (18P), raw/synthetic (38P), raw/corrections (3P), and raw/cross-session (4P). But on raw/ci, it still produced 60 inconclusives and 39 hard fails from 110 trajectories. That is a lot of noise from the strongest model in the batch. The report makes an observation about this that is worth pulling out: raw breadth without replay clarity is not yet evidence. The product can process broad raw corpora, but if it mostly yields inconclusive judgments, that is not the same as producing trustworthy results. Broad processing coverage is good. But broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness. What the cloud models clearly improved The cloud runs gave us much cleaner evidence than the early local-only picture: near-perfect parse reliability complete candidate generation across the corpus stronger results on golden and failures/positive stronger performance on corrections and broad raw corpora One model stood out: meta-llama/llama-3.1-8b-instruct was the strongest cost-effective model in the entire batch. On failures/positive, it produced 22 pass / 5 inconclusive / 3 fail — the best single result on the most practical extraction benchmark. On noisy, it produced 5 pass / 0 inconclusive / 0 fail — a clean sweep. On raw/opencode, it produced 18 pass — more than any other model. The report's verdict: best current cost/performance benchmark model. Not because it was the most expensive or the largest, but because it consistently outperformed gpt-4o-mini on the corpora that matter most for practical rule extraction. What better models did not fix This is the more important half of the story. Even with stronger cloud models, the system still struggled on the corpora that matter most for trust: successes, failures/negative, and nearmiss. The simplest local-vs-cloud safety view: Safety-sensitive corpus Best local outcome Best cloud outcome What it suggests successes still weak still weak replay and restraint remain product issues failures/negative none were convincingly strong none were convincingly strong stronger generation does not solve rejection quality nearmiss mixed mixed trigger specificity remains a hard problem On successes, the strongest cloud model produced 3 pass / 6 inconclusive / 11 fail. Those 3 "passes" are not a win — they are false positives. A "pass" on the successes corpus means the system extracted a rule from a trajectory that should not have produced one. Every pass on successes is a safety violation. On failures/negative, the strongest cloud model produced 3 pass / 1 inconclusive / 6 fail. Every model in the field test produced 5-6 hard fails on this corpus, and no model produced more than 3 passes. The rejection gate is weak regardless of which model drives extraction. On nearmiss, Cloud Llama 3.1 8B produced 6 pass / 1 inconclusive / 7 fail. Better than the locals, but still more failures than passes. The trigger is firing on cases that look similar but should not trigger. If cloud models had cleaned those up, I would be writing a very different article. They did not. The gap is not model quality — it is judgment quality This is the insight that changed how I think about the roadmap. Before the cloud runs, it was possible to believe the remaining weakness was mostly a model problem. Small local models, limited instruction-following, noisy output — maybe a bigger model would fix it. After the cloud runs, that explanation is dead. A stronger model with near-perfect parse reliability, running on a now-trustworthy benchmark, still produced 11 fails on successes and could not reliably reject bad candidates on failures/negative. The problem is not that the model cannot generate rules. The problem is that the product's judgment — when to extract, when to reject, when to stay silent — is not strong enough. The report identifies six specific gaps between the current state and a stronger release claim: Safety gap — the system performs too weakly on successes, failures/negative, and nearmiss. Replay-trust gap — the matcher and replay engine produce too many inconclusives, especially on raw corpora. Promotion-confidence gap — many plausible candidates, but not enough evidence they are consistently safe to promote. Human-judgment gap — the report leans on replay heuristics more than human-reviewed rule quality. Release-criteria gap — no explicit, enforced pass/fail thresholds tied to safety-sensitive corpora. Operational-comparison gap — we know which models are promising, but need a crisper policy for which model class qualifies as authoritative evidence. The first three are the real release blockers. The others matter, but safety, replay trust, and promotion confidence are what define whether the product can honestly claim v0.1.0 field-test success in the strongest sense. What I learned from this Better models improve the ceiling, not the floor. The cloud models lifted the best results — more passes on failures/positive, cleaner noisy handling, stronger raw corpus extraction. But they did not lift the worst results. successes and failures/negative stayed weak. That tells you the floor is a product problem, not a model problem. A "pass" on a safety corpus is not always a win. On successes, a pass means the system extracted a rule from a trajectory that should not have produced one. Counting passes without reading the corpus label gives you a false sense of progress. You have to know what each corpus is testing before you interpret the number. Inconclusives are the silent killer. gpt-4o-mini produced 248 inconclusives out of 394 candidates — 63% of its output was uncommitted. Those inconclusives did not show up as failures, so they did not trigger alarm. But they also did not produce usable rules. A model that is 63% inconclusive is not producing 77 passes. It is producing 77 passes and 248 maybes. Maybes do not build trust. And the report is clear that some of those maybes are a replay-engine limitation, not a model limitation — the matcher is the named #1 bottleneck. But on the safety corpora, inconclusives are a different signal: they mean the model extracted something from a trajectory where it should have stayed silent, and the matcher could not cleanly reject it. Those are model problems hiding behind engine weakness. You have to split the inconclusive bucket by corpus before you know which ones are the matcher's fault and which ones are the model's. The field test succeeded even though the product did not fully pass. This sounds contradictory but it is not. The field test succeeded because it forced the system through realistic comparisons, exposed tooling defects, validated that fixes improved signal, and produced a coherent map of what remains. The product improved materially. The product is not yet fully field-test complete by a strict release gate. Both of those things are true. That is a good outcome for an honest engineering report. Open questions Would a much larger model (70B, 405B) close the safety gap, or is the successes problem fundamentally about replay logic, not extraction quality? The 8B results suggest the latter, but we have not tested it. The failures/negative corpus is small (8 trajectories per model). Would a larger negative corpus surface more rejection failures, or is 8 enough to see the pattern? The current results are already weak enough to be concerning at this scale. gpt-4o-mini produced zero fails on golden (6P / 4I / 0F) but 14 fails on successes (0P / 6I / 14F). That is a striking contrast. Is the model genuinely good at extraction and genuinely bad at restraint, or is the successes corpus testing something the model was never designed to handle? The report recommends safety-first quality gates: promote only when successes, failures/negative, and nearmiss support the claim. What happens to the promotion rate if we enforce that? If it drops to near zero, that tells us the current candidate pool is not safe enough. If it stays nonzero, that gives us a credible set of rules to build on. Why this is actually good news If the cloud models had solved everything, the conclusion would have been easy: just use a better model. That would be convenient, but it would teach us less. What happened instead was better. The cloud runs improved the benchmark enough to expose the real work that remains. They showed that stronger models help a lot, but they also showed that the core challenge now lives in replay trust, safety, and promotion confidence. That is exactly what a serious field test should surface. The cloud models made the product look better. They also made its remaining weaknesses harder to deny. That is why we ran them. CauterRule v0.1.0 is released. The full all-model comparison — local vs cloud, curated and raw corpora, winner-by-corpus view, and model recommendations — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes