4,768 LLM Runs, Zero Lost Sweeps: Hardening a Field-Test Runner for Timeouts, Hangs, and Cost

Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization. Here is the failure mode that never makes it into a paper. You launch a 4,768-run benchmark. It churns for an hour. Then one request to the model provider simply never returns — no exception, no error, no timeout. The worker is blocked. The queue drains to zero throughput. You have not lost one trajectory. You have lost the sweep, and the API spend it already consumed. v0.3.0 is the first CauterRule field test that finished. The reason is not a better prompt. It is five small guards in the runner. The assumptions we started with (all of them wrong at scale) Our original runner encoded three assumptions that hold at 50 runs and fail at 4,768: Assumption 1: the provider will always respond. Assumption 2: a timeout is a transient error, so retry it. Assumption 3: a failed trajectory should be retried until it succeeds. At scale, assumption 1 produces infinite hangs, assumption 2 doubles the cost of a request that was already dead, and assumption 3 turns one bad input into an unbounded retry loop. None of these are model problems. They are harness problems — and they compound. The scale that breaks them: 40 corpora × 2 models = 4,768 runs, roughly 2,384 trajectories per model, several LLM calls per trajectory. The fix: five guards (#713) None of this is clever. It is the set of things you must do before you can trust a large sweep: Guard Implementation Failure it removes Per-trajectory timeout create(..., timeout=120) A hung call blocks a worker forever Non-retryable timeouts classify timeout as terminal Retrying a dead request multiplies cost Token cap max_tokens=4096 Runaway generations blow latency and spend Quarantine CAUTERULE_QUARANTINE_IDS Known-bad IDs are skipped, not retried Cancel on shutdown executor.shutdown(wait=False, cancel_futures=True) Interrupt hangs on queued work The shape of the concurrency fix: with ThreadPoolExecutor(max_workers=6) as executor: futures = {executor.submit(run_trajectory, t): t for t in trajectories} ... # on KeyboardInterrupt / SIGTERM: executor.shutdown(wait=False, cancel_futures=True) The principle: one bad request costs one trajectory, not the sweep. The data What the hardening bought, concretely: Metric Value Trajectory-runs completed 4,768 Models 2 (gpt-4o-mini, llama-3.1-8b) Corpora 40 Safety gate dropped (per model) 580 trajectories LLM calls avoided by the gate (per model) 1,160 Safety silence (successes / negatives) 60/60 · 60/60 Adversarial promotions 0 The safety number is only credible because every corpus finished. A benchmark that completes its easy corpora and dies on the hard ones is measuring the corpora it could finish. Bonus: cost became a first-class metric Once runs stop dying, you can afford to measure spend. v0.3.0 captures token usage on every call and prices it with real provider rates: @dataclass class LLMResponse: text: str prompt_tokens: int = 0 completion_tokens: int = 0 Model Input ($/1M) Output ($/1M) gpt-4o-mini 0.15 0.60 llama-3.1-8b 0.06 0.06 The SDK was already returning usage on every response. We were discarding it. Adding two fields turns "what does 1,000 trajectories cost?" from a shrug into a measured answer — the question that actually gates adoption. What worked The sweep completed end to end. 40 corpora, 2 models, 4,768 runs. Fail-soft held. Timeouts and quarantine dropped broken runs, not hard cases — the results stayed comparable. Cost transparency fell out for free. A few lines of token accounting, using data already on the wire. A single hung call is now a rounding error. The failure mode that motivated the work is gone. What didn't work Local models did not survive contact with scale. The same hardening made it undeniable that the local OMLX setup was too slow and hung on raw/ci. The field test went cloud-only. The "ship on local models" path from the v0.1.0 report did not hold at 40 corpora. We built measurement tools and didn't run the protocols. Cross-session repeat-failure reduction and human-vs-replay agreement both have scripts (cross_session.py, human_agreement.py) and runner flags. Neither protocol was executed. A flag that was never pulled is not evidence. We didn't instrument the drops. The runner discards timed-out trajectories cleanly, but we did not log the timeout distribution. The shape of the failure — where and how often calls hang — is itself data we threw away. Questions we still can't answer Is 120s the right timeout, or does it truncate legitimately slow multi-step extractions? We never measured how close normal calls come to the ceiling. Do the timeouts cluster in raw/ci, or are they spread evenly? We can't say without per-drop logging. Does quarantine mask a provider regression? Skipping known-bad IDs is safe for a one-off sweep; in a continuous loop it could hide a systematic failure. What is the actual $/1k-trajectory number per corpus, not just per model? We have per-model pricing but haven't published per-corpus costs. What I learned Throughput is a product feature for evaluation. Your benchmark's quality ceiling is whether it finishes. A harness that can't survive a hung call can't run the corpora where the interesting failures live. Fail-soft beats fail-hard. Retries that assume transience are a cost bug waiting to happen. Classify failures as terminal vs. transient explicitly. Capture usage while you're already holding the client. Token accounting is trivial when you own the call site and impossible when you don't. Do it from day one. Building the instrument is not taking the reading. We shipped cross-session and agreement tooling and left both protocols unrun, so those questions are still open. Tooling is not measurement. The broader lesson If you run large evaluations against a flaky external API, the infrastructure is not a distraction from the science — it is part of the science. The most valuable code in CauterRule's v0.3.0 field test was not a matcher or a prompt. It was five guards that let 4,768 requests finish, plus two dataclass fields that let us price them. That is what turned a benchmark that kept dying into one you can trust — and, conveniently, one that can tell you what it costs. References CauterRule v0.3.0 release notes v0.3.0 field test report Cost measurement (token capture + per-model pricing) Field test plan (corpora, models, runner flags) Generated results (all 40 corpora) PRD — architecture User guide · Changelog CauterRule v0.3.0 is released. The methodology, hardening notes, and cost report are in the field test report and the cost measurement doc. The repo is public. Install with pip install cauterule. Changelog · Release notes