Your Agent Returned 200 and Did Nothing: Detecting Silent Failures with Token Counts

Your agent returned HTTP 200, zero errors in the logs, and then did nothing. The failure wasn't loud — it was silent. This happens more than you'd think. An agent invokes a model, gets back a response, processes it, returns success. But somewhere in that flow the actual output — the thing that was supposed to happen — vanished. No exception. No timeout. Just an empty response where a decision or action should have been. Standard monitoring catches loud failures: connection timeouts, rate limits, exceptions. It misses the silent ones. Here's why, and how to spot them. The three signals most monitors ignore When an LLM agent runs, three things happen: The HTTP call completes (status 200 or 400+) Tokens are consumed (input, then output) Actual work occurs — a response is generated, a decision is made, text comes back Most observability tools watch #1. Some logging catches parts of #2. Almost nobody watches #3 in a way that actually matters. The mechanical truth: HTTP 200 and a nonzero token count do not guarantee real output. The model refuses silently An agent asks Claude (or GPT, or any model) to do something that brushes up against a boundary — political, violent, ambiguous. The refusal policy triggers. It comes back: HTTP 200 input_tokens: 450 output_tokens: 127 response: "" // or a short refusal like "[No response]" The call succeeded. Tokens were spent. Nothing usable came out. If your downstream logic checks if response: do_something(), you've got a silent failure. The model generates emptiness Less common, but real: the model returns a completion that's all whitespace or filler with no actual content. Output tokens go up. Meaning doesn't. HTTP 200 input_tokens: 300 output_tokens: 89 response: " \n\n " // whitespace, no content Tokens spent, call succeeded, nothing happened. Tool use fails quietly An agent calls a tool — database lookup, API call, calculation. The tool errors or returns nothing. The model, correctly, generates a response explaining it couldn't do the thing. Except your downstream code expected a value, not an apology. HTTP 200 input_tokens: 520 output_tokens: 212 response: "I attempted to fetch the user but the database returned no match." expected: { user_id: 12345, status: "active" } The agent succeeded at speaking. It failed at acting. Why token count is the canary Output tokens measure how much the model actually generated, independent of quality. A genuinely empty response will have an output token count near zero. A response that merely looks empty — but has some content, even noise — will burn tokens. That's why token count matters more than HTTP status: HTTP 200 tells you the infrastructure worked. Output tokens > 0 tells you the model generated something. Actual content length tells you if there's signal or just noise. Rough heuristic: if output tokens are suspiciously low for the task, or nonzero but the response is empty/whitespace, you're looking at a silent failure. A simple detector def detect_silent_failure(agent_run): http_ok = agent_run.status_code == 200 tokens_spent = agent_run.output_tokens > 0 has_content = len(agent_run.response.strip()) > min_threshold # e.g. 10 chars if http_ok and tokens_spent and not has_content: return "SILENT_FAILURE" if http_ok and tokens_spent < expected_tokens - threshold: return "UNDERSHOOTING" return "OK" The key is comparing actual output against token count. If a model spent 150 output tokens and your response is empty, something broke between generation and delivery. Why it's worth catching A crash tells you something broke — you debug it. A "success" that produced nothing is worse: it quietly corrupts pipelines, skips records, or leaves a user waiting on a response that's never coming. HTTP status and token count both look fine, so plain logging won't catch it. You need something that checks whether the response actually matches what was supposed to happen. In practice If you're running agents and not instrumenting for this, you're flying blind. Three things help: Log output token count alongside every response. Put it in your metrics, not just your logs. Flag runs where tokens are spent but response length is suspiciously small. Trace downstream failures back to the source — did the agent really produce output, or did it just look like it did? The green light doesn't mean it's working. The token count tells you if anyone's actually home.