Testing AI Agents in TypeScript Without Calling the Model

Book: AI That Acts The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub Ask someone how they test their agent and the answer is usually "we run it and see." Which is fair, because the model is nondeterministic, but it quietly concedes that the loop around the model is untested too, and that is where the expensive bugs live. The runaway loop, the missing tool_result, the budget check that runs after the spend, the retry that re-sends a completed side effect: none of those are model behaviour. They are your control flow, and control flow is testable. The seam You need exactly one: the thing that produces a model response. export interface ModelClient { create(params: MessageCreateParams): Promise; } export async function runAgent( task: string, ctx: Ctx, limits: Limits, client: ModelClient = realClient, ) { /* ... */ } A default parameter keeps production call sites unchanged and makes every test able to substitute. That is the entire refactor. A fake that plays a script type Scripted = | { text: string } | { tools: { name: string; input: unknown }[] }; export function scriptedClient(script: Scripted[]) { let i = 0; const calls: MessageCreateParams[] = []; return { calls, get turns() { return i; }, async create(params: MessageCreateParams): Promise { calls.push(params); const step = script[Math.min(i, script.length - 1)]; i++; return "text" in step ? message([{ type: "text", text: step.text }], "end_turn") : message( step.tools.map((t, n) => ({ type: "tool_use" as const, id: `tu_${i}_${n}`, name: t.name, input: t.input, })), "tool_use", ); }, }; } const message = (content: ContentBlock[], stop_reason: string): Message => ({ id: "msg_test", type: "message", role: "assistant", model: "test", content, stop_reason, stop_sequence: null, usage: { input_tokens: 100, output_tokens: 50 }, } as Message); Clamping to the last step (Math.min) is what makes infinite-loop tests possible: a one-element script of a tool call repeats forever, exactly like a stuck model. Recording calls is what makes assertions about your behaviour possible — what you sent, in what order, with what history. The tests that earn their keep The runaway loop. This is the failure that produces the incident reports, and it takes four lines to cover. it("stops at the turn cap when the model never finishes", async () => { const client = scriptedClient([{ tools: [{ name: "search", input: { q: "x" } }] }]); const out = await runAgent("go", ctx, { maxTurns: 5, maxCostUsd: 99 }, client); expect(client.turns).toBe(5); expect(out.status).toBe("turn_limit"); }); The budget check happens before the spend. Ordering bugs here are invisible in production until the bill. it("does not call the model when the budget is already exhausted", async () => { const client = scriptedClient([{ text: "hi" }]); const out = await runAgent("go", ctx, { maxTurns: 10, maxCostUsd: 0 }, client); expect(client.turns).toBe(0); expect(out.status).toBe("budget"); }); toBe(0) is the assertion. A loop that checks the budget at the bottom passes a status check and fails this one. Every tool_use gets a tool_result. A missing pair is a 400 from the API, and it only shows up on multi-tool turns. it("returns one result per tool call, matched by id", async () => { const client = scriptedClient([ { tools: [ { name: "search", input: { q: "a" } }, { name: "get_order", input: { id: "bad" } }, // will fail ] }, { text: "done" }, ]); await runAgent("go", ctx, limits, client); const second = client.calls[1].messages.at(-1)!; const results = second.content as ToolResultBlockParam[]; expect(results).toHaveLength(2); expect(results.map((r) => r.tool_use_id).sort()) .toEqual(["tu_1_0", "tu_1_1"]); expect(results.some((r) => r.is_error)).toBe(true); }); Inspecting client.calls[1] is the trick — you are asserting on the request your loop built, which is the thing that would have been malformed. A failing tool does not kill the run. it("continues after a tool throws", async () => { const ctx = ctxWith({ search: async () => { throw new Error("upstream 503"); } }); const client = scriptedClient([ { tools: [{ name: "search", input: { q: "x" } }] }, { text: "I could not search, but here is what I know." }, ]); const out = await runAgent("go", ctx, limits, client); expect(out.status).toBe("complete"); expect(client.turns).toBe(2); }); Internal errors do not reach the model. A stack trace in a tool_result gets summarised and can be quoted to a user. it("does not leak internal error detail to the model", async () => { const ctx = ctxWith({ get_order: async () => { throw new Error("pg: relation \"orders_v2\" does not exist at 10.0.0.4:5432"); }}); const client = scriptedClient([ { tools: [{ name: "get_order", input: { id: "1" } }] }, { text: "done" }, ]); await runAgent("go", ctx, limits, client); const sent = JSON.stringify(client.calls[1]); expect(sent).not.toMatch(/10\.0\.0\.4|orders_v2|pg:/); }); Context does not grow without bound. it("compacts once the window exceeds its budget", async () => { const client = scriptedClient([{ tools: [{ name: "big", input: {} }] }]); await runAgent("go", ctxWithHugeToolOutput(), { maxTurns: 8, ...limits }, client); const sizes = client.calls.map((c) => JSON.stringify(c.messages).length); expect(Math.max(...sizes)).toBeLessThan(200_000); }); That one catches the quadratic-cost bug — every turn resending an ever-larger array, which is invisible in a two-turn manual test and expensive at eight. Fake the tools too export function ctxWith(overrides: Partial): Ctx { return { ...baseCtx, tools: { ...defaultFakeTools, ...overrides }, }; } Tool fakes let you drive the interesting paths: empty results, 429s, timeouts, malformed payloads. Those are the branches where loop logic goes wrong, and they are hard to trigger against a real upstream on purpose. Where this stops These tests say nothing about answer quality, tool selection, or whether the prompt is any good. That is what evals are for, and evals are slow, cost money, and are statistical rather than binary. The split that works: dozens of fast deterministic tests for the loop, running on every commit; a small graded eval suite for behaviour, running on a schedule or before release. Most teams have the second and not the first, which is backwards — the loop is where the bugs that cost money live, and it is the half you can actually pin down. The starting five If you write nothing else: turn cap fires, budget checked before the call, one tool_result per tool_use matched by id, a throwing tool does not end the run, internal errors do not reach the model. Half an hour, no API key, and they cover the failures that otherwise get discovered in production. If this was useful AI That Acts builds the agent loop with its seams in the right places — a substitutable client, tool fakes, and the guards these tests are written against. Graded eval suites are book five. The series is at xgabriel.com/ai-in-typescript.