How to Test an AI Agent: Start With Error Analysis, Not Metrics

Your agent passed the demo. Does it still work on the 500th request? Agents fail differently from normal software. A normal bug throws an exception. An agent bug returns a confident, well-formatted, wrong answer with HTTP 200: the right tool with the wrong ID, an extra loop, a step quietly skipped. Nothing alerts you, and the only witness is an unhappy user. The good news is that you already know the fix. It's the testing discipline you practise every day, adapted to a component that is non-deterministic and takes many steps. Traces replace stack traces, eval datasets replace test fixtures, and graders replace assertions. Why agents are hard to test Four properties make agents different: Non-determinism. The same task can produce different tool sequences on different runs. One green run proves the agent can succeed, not that it reliably will. Multi-step trajectories. The answer can be right for the wrong reasons. You have to test the path, not only the destination. Compounding errors. An agent that is 95% reliable per step succeeds only about 60% of the time over ten steps (0.95 to the power of 10). Silent failure. Fluent text with a 200 status. Monitoring has to inspect content, not just status codes and latency. Step 1: measure flakiness before anything else The cheapest first test needs no graders at all. Run each task several times and see how consistent the agent really is: from collections import Counter TASKS = ["Refund order A123", "Change my email to a@b.com", "Where is my parcel?"] N = 5 for task in TASKS: outcomes = [] for _ in range(N): trace = run_agent(task) # your agent, instrumented tools = tuple(c.name for c in trace.tool_calls) outcomes.append((trace.success, tools)) passes = sum(ok for ok, _ in outcomes) paths = Counter(t for _, t in outcomes) print(f"{task!r}: {passes}/{N} passed, {len(paths)} distinct tool paths") If one task shows 3/5 passes and four different tool paths, you have found your first reliability problem. Step 2: read your traces before you write a single metric This is the step most teams skip, and the one that matters most. Teams that ship reliable agents start by reading 50 to 100 real traces and writing down what went wrong. The process is borrowed from qualitative research: Collect traces. Pull 50 to 100 real runs from production or staging. Open coding. For each trace, write a free-form note on the first failure you see. Later failures are often side effects of the first. Axial coding. Group your notes into a taxonomy of 5 to 8 specific failure modes. Count. Tally each mode. The top two or three are where you write evals. A spreadsheet is enough. A typical result for a support agent might be: wrong customer looked up (14), refund issued without a policy check (9), answered from memory instead of the knowledge base (7). Those three lines are your eval roadmap. Stop when new traces stop revealing new failure modes, which usually happens within about 100. An LLM can help cluster your notes, but it never replaces the reading. Step 3: trace everything You can't do error analysis without traces. A trace is the agent's flight recorder: every model call, tool call and retrieval, with tokens, cost and timing. It's a tree of spans, with one invoke_agent root and chat and execute_tool children. Instrument against the OpenTelemetry GenAI conventions for portability, but pin versions, because they are still marked as in development. Tag every run with a prompt version and an agent version, so a regression can be tied to the change that caused it. Langfuse and Arize Phoenix are open source; LangSmith and Braintrust are managed; Datadog, Honeycomb and Grafana let you keep agent traces next to the rest of your telemetry. The tool matters less than the habit of reading traces every week. Step 4: build a small eval dataset from real failures Start with 20 to 50 tasks drawn from the failures you found, and grow the set every week. Every task needs a success criterion two engineers would agree on. Include negative cases too, such as a refund request that should be refused, or you will optimise the agent into acting when it shouldn't. If you generate synthetic inputs, generate them along explicit dimensions (intent × persona × difficulty) rather than asking for "50 questions". Let an LLM draft the inputs, but write the expected outcomes yourself. Step 5: grade with code wherever you can Use code for anything checkable: final state, which tools were called, their arguments, output format, step and cost budgets, safety rules. Final-state checks are the strongest graders for agents that take actions. Did the refunds table end up with exactly the right rows? Two rules keep code graders from becoming brittle. Assert what must and must not happen, but never hard-code one exact tool sequence. And give every trial a fresh, isolated environment, or your scores measure test pollution instead of the agent. Step 6: use LLM judges narrowly, and calibrate them For things code can't check, like "did every policy claim come from the knowledge base?", an LLM judge can scale human judgement to thousands of traces. But judges have known biases: position, verbosity and self-preference. Design around them: one narrow judge per failure mode, with a binary pass or fail reasoning before the verdict "length is not a virtue" stated in the rubric a judge from a different provider than the agent Then calibrate the judge like a classifier against traces a human has labelled. Measure its true-positive and true-negative rates, not raw agreement. A judge with a 60% true-negative rate misses four in ten real failures. Step 7: track the right numbers Report four numbers per suite: pass@1, pass^k, cost per task and latency per task. pass@k means at least one of k attempts succeeded; pass^k means all k did. A customer-facing agent needs pass^k, because a customer gets one attempt. Run each task 3 to 5 times, and watch median and p95 cost and latency: long tails usually reveal loops. The north star Look at your data before you build anything. The rest — datasets, graders, judges and CI gates — only works if it targets the failures your agent actually has. Originally published on TechBrosEbooks. This article is adapted from the ebook Is Your AI Agent Actually Working?, which goes further with OpenTelemetry instrumentation code, a reusable grader library, judge calibration, evals in CI/CD and production, and an end-to-end example of fixing a real support-triage agent.