One Passing Agent Run Is Not a Release Signal

I can make almost any agent change look good with one carefully chosen prompt. Run it once. Watch the right tool fire. Read a polished answer. Record the demo. Ship it? Not yet. A single trace is excellent evidence for one execution: what ran, what failed, how long it took, and whether declared invariants held. A release decision asks a different question: Does this change behave acceptably across the representative cases we care about? That shift—from run to set—is where agent testing starts to resemble engineering instead of demonstration. Give each evidence layer one job In agent-inspect@6.17.6, I use three local, deterministic layers: Layer Question Suite Did each named case satisfy its expected checks? Cohort What moved between baseline and candidate groups? Gate Should CI accept the recorded evidence? representative traces | v suite --------> named case results | v baseline vs candidate -> cohort deltas | v gate --------> CI exit code + evidence These commands read persisted traces. They do not rerun the agent, invoke a model, or decide whether prose is semantically good. Start with named cases, not random traffic A compact suite makes the review set explicit: { "name": "refund-release", "traces": "./.agent-inspect", "cases": [ { "id": "eligible-order", "runId": "refund-eligible", "requireTools": ["lookup_order", "refund_order"] }, { "id": "unknown-order", "runId": "refund-unknown", "forbidTools": ["refund_order"] }, { "id": "approval-required", "runId": "refund-approval", "requireTools": ["request_approval"] } ] } Validate the configuration and run it: npx agent-inspect suite validate \ --config agent-inspect.suite.json npx agent-inspect suite run \ --config agent-inspect.suite.json \ --markdown The suite distinguishes a violated expectation from missing evidence. If refund-unknown does not exist, the case is skipped with a diagnostic; it is not silently treated as a pass. An all-skipped suite cannot become a green release signal. That distinction sounds small until CI says “passed” for a case that never ran. The set is part of the test The most important suite question is not “Did it pass?” It is “Why are these the representative cases?” For a refund agent, I would begin with: an ordinary eligible order; an unknown order that must not trigger a refund; an amount requiring approval; a missing-data path; a retryable dependency failure; and a non-retryable tool failure. Version the suite beside the application. When product behavior intentionally changes, update the evidence set and expectations in the same review. A useful review matrix keeps scenario coverage and evidence type visible: Scenario Structural check Outcome check Semantic check Eligible order lookup before refund refund record exists explanation is accurate Unknown order refund tool forbidden no mutation occurred refusal is useful Approval required approval tool present request queued message explains next step No single column can substitute for the other two. A structurally correct run can still answer badly; a fluent answer can still hide an unauthorized mutation. This is also where “prove the suite can say no” becomes useful. Keep at least one known-bad trace or mutation that must fail. A check that has never rejected evidence and a check that silently stopped running can produce the same green icon. Compare behavior as a cohort For a prompt, model, tool-schema, or orchestration change, label captured runs: import { inspectRun } from "agent-inspect"; await inspectRun("refund-agent", runScenario, { traceDir: ".agent-inspect", metadata: { cohort: "candidate", scenario: "eligible-order", model: "approved-model", }, }); Then compare baseline and candidate traces: npx agent-inspect cohort \ --dir .agent-inspect \ --baseline before \ --candidate candidate \ --cohort-key cohort \ --group-by model \ --metric errorRate,duration,toolChoice,observationFailure A cohort report preserves the run count and the actual aggregate values. Imagine this synthetic result: before / approved-model (10 runs) error rate: 0% average duration: 820ms dominant tool: lookup_order candidate / approved-model (10 runs) error rate: 10% average duration: 970ms dominant tool: refund_order That is a reason to investigate, not a statistically universal conclusion. The CLI can describe the recorded groups; it cannot make a small, biased fixture set representative. A positive delta is not automatically bad Suppose the candidate makes one extra tool call. That might be waste—or a newly required authorization check. Suppose duration improves by 20%. That might be a better path—or the candidate skipped retrieval. Suppose the dominant tool changed. That might be a regression—or the exact migration you intended. Structural metrics describe what changed. Product assertions and semantic evaluation decide whether the change is acceptable. Keep those judgments separate so the report does not pretend to know your domain. Turn credible evidence into a fail-closed gate Once cases and thresholds reflect actual release policy, add a gate: npx agent-inspect gate \ --suite agent-inspect.suite.json \ --format github \ --output ./agent-gate-artifacts \ --evidence-on fail \ --evidence-profile share Or gate a directory directly: npx agent-inspect gate \ --dir .agent-inspect \ --max-error-rate 5 \ --forbid-tool delete_account Gate exit codes distinguish a policy failure from invalid configuration, unreadable traces, and unsupported output formats. Evidence generation does not suppress the original gate result. That behavior matters because agent tooling is especially vulnerable to accidental fail-open configurations. Version 6.17.5 strengthened empty-check handling, rule-execution evidence, observed-outcome requirements, and requiredOrder semantics; 6.17.6 preserves those behaviors while hardening packaging and ingest boundaries. What a green gate actually means A passing gate means the encoded checks passed for the inspected evidence set. It does not prove: the cases represent production traffic; the final answers are correct; the model will repeat the behavior; the thresholds are good; or the agent is safe in production. That is not a weakness unique to agent tests. It is the normal boundary of evidence-based release decisions. The remedy is not one giant magical score; it is several checks with clear jobs. My smallest credible release loop capture named cases -> run deterministic suite -> compare baseline/candidate cohorts -> inspect semantic quality separately -> make CI fail closed -> preserve the evidence The pinned suites, cohorts, and gates documentation labels these surfaces Beta, and the cohort recipe includes a deliberately regressing fixture. One good run can prove that a path exists. It cannot prove that a release is ready. What is the smallest set of agent cases you would require before approving a change?