Reading Agent Transcripts for Real Observability, Without Instrumentation

When you fan out ten subagents and each one returns a confident wall of text, the bottleneck stops being generation and starts being verification. A model is good at producing candidates and bad at knowing what counts as proof. So the real question is not "did the agent answer" but "which of these answers can I trust." And the honest answer, most days, is that you do not know, because nobody reads the transcript once the work is done. I want to make the case that you can get most of the observability you actually need by reading the transcript carefully, without adding a single line of instrumentation. I built a small tool called agentrace to do this reading for me, and I will use it to make the ideas concrete. But the tool is not the point. The point is learning to read a run the way you would read a junior's pull request: looking for the specific places where confidence and evidence have come apart. What a transcript already contains Here is the part people miss. If you run agents through Claude Code, every session is already written to disk as a .jsonl file, including every delegation and its result. The data is there whether or not you planned to trace anything. You do not need an SDK or a wrapper. You need to open the file and know what to look for. That reframes observability from "something I should have set up beforehand" into "something I can do to the run I already have." A transcript gives you three things reliably: what you asked, what came back, and how long it took. Everything useful I am about to describe is built out of just those three. What it can tell you Silent failures. The cheapest and most embarrassing one. An agent returns nothing, or returns four words, or dies on a session limit mid-sweep. It is trivial to detect and easy to miss when you are skimming a wall of output, and it silently means the work did not happen. When I ran agentrace against the 34MB session that motivated it, 7 of 152 runs had errored this way. Nobody had noticed, because a short report looks a lot like a done report. Polite giving up. "I was unable to find any open roles" reads like an answer if you skim. It is not one. The agent did the linguistic shape of a conclusion without the substance. Grep your own transcripts for "I was unable to," "I could not determine," "no results were found." Each hit is a place where you probably moved on believing the question was settled. Hedged claims that get flattened. This is the one that bit hardest. An agent said a company "appears to be" hiring. That hedge was honest and correct. But by the time the finding reached a decision, "appears to be" had quietly become "is." The hedge is not the bug. Flattening it downstream is the bug. Reading the transcript lets you catch the hedge while it is still a hedge, before your next step launders it into a fact. Words like "appears to be," "likely," "probably," "I believe," "could not verify" are load bearing, and they need to survive into whatever consumes the result. Absence treated as evidence. An agent concluded a company was not hiring because an API returned an empty list. That API returns empty with HTTP 200 for accounts that do not exist. Absence of data is not evidence of absence, and the whole finding lives in that difference. When a run concludes "there are no X," the transcript is where you check whether the negative was real or a null signal: a 200 with an empty body, a 404 on a valid resource, a page that only renders in JavaScript. Claims with no verification behind them. An agent that lists twenty URLs it never opened is doing autocomplete, not research. The tell is in the text: many links, no mention of fetching, checking, or confirming any of them. You are reading for the gap between how much is asserted and how much was actually touched. Your own bad prompts. This is the finding that changed how I work. Twelve of the flagged runs in that session were not the model failing. They were prompts where I forgot to say what the output should be. A task with no definition of done cannot be verified, because I never really asked the question. If the prompt never specifies a shape, the result is whatever the agent felt like returning, and I have no standing to complain that it is hard to parse. Most agent tooling assumes the model is the problem. The transcript kept telling me the caller was. What it cannot tell you Here is the honest caveat, and it matters more than any check. Every one of these signals is a heuristic over text. It tells you what to go read. It does not tell you what is true. A transcript cannot confirm that a verified URL says what the agent claims it says. It cannot tell you a confident, well-formatted, contract-satisfying answer is actually correct, because being fluent and being right are independent. Reading the transcript narrows where you spend your attention. It does not replace judgment, and treating a green run as a proof is just a more sophisticated version of the mistake we started with. That constraint shapes everything. A checker that cries wolf gets switched off, which is worse than no checker at all. So the discipline is to keep the signals conservative and to prefer "here is a place worth looking" over "here is a verdict." I learned this the slow way. My prompt check used to fire on anything under 200 characters, until it flagged "run the suite and report every failing test as node ids with its assertion message," which is 113 characters and perfectly verifiable. Length was never the defect. Being short and never saying what done looks like is. Tightening that took the noise down without losing a single real finding, and the noise is what determines whether you keep reading the reports at all. How to actually do this You do not need my tool. Open a session file and grep for the hedge words, the giving-up phrases, the empty and near-empty results, the negative conclusions, the runs that took suspiciously long. Read the prompts you wrote and ask whether each one stated a definition of done. Do that by hand a few times and you will feel the pattern: observability for agents is mostly the habit of reading the run against the claim, and noticing where they diverge. Against the session that motivated it, this reading flagged 36 of 152 runs. Not because the model was broken 36 times, but because 36 times the confident output and the actual evidence had come apart, and only reading closed the gap. If you want a tool that does the reading for you, the checks and the reasoning behind each one are at github.com/AgentPostmortem/agentrace. Take the ideas even if you skip the tool.