Fast Agent Quality Gates: Deterministic Rules Over LLM Judges

Deterministic agent tests become much more valuable when they are expressed as reusable quality gates rather than one-off assertions scattered across test files. A gate answers a narrow engineering question: Did the run validate before writing data? Did retries stay within policy? Was token usage recorded and within budget? Did every started span finish? The result should be stable, fast, and specific enough that a developer knows what to fix. LLM judges still have a role in semantic evaluation. They should not be the only thing standing between a structural agent regression and production. What Makes a Good Gate? A practical quality gate has five properties: Deterministic: The same normalized trace produces the same result. Narrow: It evaluates one explicit contract. Actionable: Failure identifies the rule, evidence, and expected state. Versioned: Rule and trace-schema changes can be reviewed. Portable: It can run locally, in CI, and against replayed fixtures. “Answer quality was 6/10” is a signal, but it is not a narrow engineering contract. “The write tool ran before authorization completed” is. Start With a Stable Trace Contract The gate engine should consume normalized metadata rather than framework-specific callback objects. type StepKind = 'run' | 'model' | 'tool' | 'retrieval' | 'policy' | 'fallback'; type TraceStep = { id: string; parentId: string | null; sequence: number; name: string; kind: StepKind; status: 'ok' | 'error' | 'blocked' | 'cancelled'; attempt?: number; inputTokens?: number; outputTokens?: number; durationMs?: number; metadata?: Record; }; type AgentTrace = { schemaVersion: 1; fixture: string; status: TraceStep['status']; steps: TraceStep[]; }; Normalize volatile fields before evaluation. Random IDs can remain for parent-child checks, but timestamps, temporary paths, request IDs, and raw payloads should not influence deterministic results. Define a Rule Interface Rules should return structured evidence instead of throwing assertion errors directly. type Severity = 'error' | 'warning'; type RuleResult = { ruleId: string; severity: Severity; passed: boolean; message: string; evidence?: Record; }; type TraceRule = { id: string; version: number; severity: Severity; evaluate(trace: AgentTrace): RuleResult; }; function result( rule: TraceRule, passed: boolean, message: string, evidence?: RuleResult['evidence'], ): RuleResult { return { ruleId: `${rule.id}@${rule.version}`, severity: rule.severity, passed, message, evidence, }; } Versioning a rule makes baseline changes explicit. If the meaning of max_model_calls changes, reviewers can see that the policy changed rather than assuming the agent regressed. Gate 1: Required Steps function requireSteps(required: string[]): TraceRule { return { id: 'required_steps', version: 1, severity: 'error', evaluate(trace) { const actual = new Set(trace.steps.map((step) => step.name)); const missing = required.filter((name) => !actual.has(name)); return result( this, missing.length === 0, missing.length === 0 ? 'All required steps ran' : `Missing required steps: ${missing.join(', ')}`, { missingCount: missing.length }, ); }, }; } Required-step rules work well for validation, retrieval, policy checks, and mandatory cleanup. Do not require every implementation detail; gates should protect behavior that matters to users, cost, safety, or correctness. Gate 2: Causal Order For sequential dependencies, compare the recorder’s monotonic sequence. For concurrent work, assert parentage instead of completion order. function requireOrder(before: string, after: string): TraceRule { return { id: `order:${before}:${after}`, version: 1, severity: 'error', evaluate(trace) { const left = trace.steps.find((step) => step.name === before); const right = trace.steps.find((step) => step.name === after); if (!left || !right) { return result(this, false, 'Cannot evaluate order: step missing'); } return result( this, left.sequence < right.sequence, left.sequence < right.sequence ? `${before} occurred before ${after}` : `${after} occurred before required dependency ${before}`, { beforeSequence: left.sequence, afterSequence: right.sequence }, ); }, }; } Authorization-before-write and retrieval-before-generation are good causal gates. Ordering every trace step creates brittle tests and blocks harmless parallelization. Gate 3: Forbidden Work After a Block const noExternalWorkAfterBlock: TraceRule = { id: 'no_external_work_after_block', version: 1, severity: 'error', evaluate(trace) { const block = trace.steps.find((step) => step.status === 'blocked'); if (!block) return result(this, true, 'Run was not blocked'); const forbidden = trace.steps.filter((step) => { return ( step.sequence > block.sequence && (step.kind === 'model' || step.kind === 'tool') ); }); return result( this, forbidden.length === 0, forbidden.length === 0 ? 'No model or tool work occurred after the block' : `External work continued after block: ${forbidden .map((step) => step.name) .join(', ')}`, { forbiddenCount: forbidden.length }, ); }, }; This is stronger than checking only the final status. A run can report blocked and still leak a model or tool call if orchestration continues incorrectly. Gate 4: Attempts and Loops Retries should include an explicit attempt field. Count attempts by operation and parent span instead of looking for consecutive names, because parallel events can interleave. function maxAttempts(stepName: string, limit: number): TraceRule { return { id: `max_attempts:${stepName}`, version: 1, severity: 'error', evaluate(trace) { const attempts = trace.steps .filter((step) => step.name === stepName) .map((step) => step.attempt ?? 1); const maximum = attempts.length === 0 ? 0 : Math.max(...attempts); return result( this, maximum { return step.inputTokens === undefined || step.outputTokens === undefined; }); if (missingUsage.length > 0) { return result(this, false, 'Model usage is missing', { missingUsageCount: missingUsage.length, }); } const total = modelSteps.reduce((sum, step) => { return sum + (step.inputTokens ?? 0) + (step.outputTokens ?? 0); }, 0); return result( this, total { return !item.passed && item.severity === 'error'; }); const warnings = results.filter((item) => { return !item.passed && item.severity === 'warning'; }); return { fixture: trace.fixture, passed: failures.length === 0, failures, warnings, results, }; } Write the report as JSON for automation and as a short Markdown summary for pull-request logs or CI annotations. Include rule versions, evidence, fixture names, and a link or path to the normalized trace artifact. Compare Baselines Without Snapshot Brittleness Exact trace snapshots are difficult to maintain. Prefer a summary of durable metrics: type TraceBaseline = { fixture: string; requiredTools: string[]; maximumModelCalls: number; maximumTokens: number; maximumAttemptsByTool: Record; }; Use both absolute and relative limits. A 50% token increase from 100 to 150 may be harmless; a 50% increase from 20,000 to 30,000 may be costly. Conversely, an absolute increase of 500 tokens is significant for a small workflow and noise for a very large one. Require intentional baseline updates in the same pull request as the behavior change. The review should explain why the new budget or tool path is acceptable. Separate Synthetic and Live Thresholds Scripted orchestration tests should use fake clocks and exact budgets. Live-model and network tests have natural variance and need broader statistical thresholds. Do not use one threshold for both. A local fixture that suddenly takes ten seconds likely indicates a bug. A real provider call crossing a narrow latency threshold once may only reflect transient infrastructure conditions. For live runs, compare rolling distributions such as median and tail latency over enough samples. Keep those trend checks outside the fastest pull-request gate unless the project has the capacity to operate them reliably. A CI Contract A provider-neutral CI job can follow this sequence: 1. Run scripted agent fixtures 2. Validate every normalized trace 3. Evaluate the configured rule set 4. Write JSON and Markdown reports 5. Exit non-zero when error-severity rules fail 6. Upload reduced trace artifacts for failed fixtures 7. Retain artifacts for a short, explicit period Use the project’s existing runtime-version file and package-manager lockfile rather than hard-coding setup details into the article or gate engine. Real-model credentials should be unavailable to the deterministic job. Run semantic evaluations in a separate job with explicit authorization, cost controls, and a slower cadence. Avoid Gate Inflation Too many brittle gates make developers ignore the system. Add a rule only when it protects a meaningful contract and has an owner. Use error for correctness, safety, or hard budget violations. Use warning for trends that need review but should not block immediately. Track warning age; a warning that never becomes actionable should be removed or converted into a real policy. When a gate fails repeatedly for accepted behavior, fix the rule or the baseline. Do not normalize permanent red CI. Final Thought Agent quality gates work best when they treat execution as an engineering artifact. A normalized trace, a versioned rule set, and an evidence-rich report can catch missing validation, unauthorized work, retry storms, cost regressions, and broken instrumentation in seconds. Use deterministic gates for contracts the trace can prove. Keep semantic judges for language and reasoning quality, where probabilistic evaluation is actually necessary. That separation makes CI faster and makes every failure easier to trust. The next article will move from rules to integrations: how adapters translate different TypeScript agent frameworks into one trace model without coupling the core to any single SDK.