Switchboard: building a tool router so your AI agent stops drowning in MCP tools

Keyword search picks the right tool for an AI agent 21% of the time. The router we built picks it 88% of the time, while cutting the tokens spent describing those tools by 99.6%. This is how we got there, including the parts we got wrong first. Architecture: Link The problem nobody talks about until the bill shows up Model Context Protocol (MCP) solved a real problem. It gave every AI agent a standard way to talk to external tools: CRMs, observability stacks, email, vector databases, internal APIs. It created a quieter problem in the process. Once you connect more than a handful of MCP servers to a single agent, you are no longer describing a tool catalog to the model. You are dumping an entire warehouse inventory into every prompt. In our own environment, three backends exposed 142 tools. Every one of those schemas, with its names, descriptions, and parameter shapes, gets tokenized and sent to the LLM on every turn, whether the user's question needs one of them or none of them. That has three costs, and only one of them shows up on an invoice. The first is token cost. You are paying to re-describe N tools you will never call, every single time. The second is selection accuracy. The more tools an LLM has to choose from in a single context, the more often it picks the wrong one, hallucinates parameters, or gets confused by two similarly-named tools from different servers. The third is operational fragility. Every new MCP server you connect makes the prompt bigger and the agent's job harder, so scaling tool count and scaling reliability end up pulling in opposite directions. We built Switchboard to decouple those three curves. The name is the metaphor: a telephone switchboard operator connects your call to the right line so you never need to know the number. Switchboard does that for tools. The host asks for what it wants in plain language, and the router works out which of 142 lines to patch it through to. Two tools instead of two hundred Switchboard sits between your AI host (Claude Code, a custom chat agent, an IDE, anything that speaks MCP) and every backend MCP server you own. From the host's point of view Switchboard is an MCP server, but instead of exposing your full tool catalog it exposes exactly two meta-tools. find_tools(request) takes the user's request in plain language and returns a small, dynamically-sized set of relevant tools: zero, one, or a handful, never a fixed top-k and never the whole catalog. invoke(tool_id, args) then calls the selected tool against whichever backend actually owns it. Everything else is the router's problem rather than the host's: which backend hosts which tool, how many servers are connected, how the catalog changes over time. How find_tools actually decides This is the part that took the most iteration, because search alone is not enough. A naive nearest-neighbor lookup either returns too much, which defeats the purpose, or misses the right tool because of vocabulary mismatch between how a user asks and how a tool is described. The retrieval pipeline runs in four stages. It starts with concurrent dense and sparse search against a Pinecone vector registry. Dense embeddings catch semantic similarity, so "send the report to finance" finds send_email_with_attachment. Sparse keyword-style search catches the exact-term matches dense embeddings sometimes miss: API names, service identifiers, error codes. Both index round-trips are independent I/O, so they run in parallel and cost one round-trip of wall-clock rather than two. Next, a cosine-similarity gate filters out anything too far from the query's intent before it ever reaches the LLM. It is cheap and deterministic, and it keeps obviously irrelevant tools from wasting judge tokens. Then an LLM judge does the expensive part properly: capability-fit selection, de-duplication across near-identical tools from different backends, ordering by execution sequence, and deciding to ask a clarifying question instead of guessing when the request is genuinely ambiguous. Finally, the result count is fully dynamic. Most routing systems force a fixed top-k. We don't. Some queries need zero tools, some need exactly one, and a few legitimately need several, so the judge decides rather than a hardcoded number. If find_tools comes back empty there is a fallback, find_more_tools, that relaxes the gate before giving up and clarifying. It is a second chance before the decision gets punted back to the user. How we measured accuracy, and why the grader is strict The headline is 85 to 90% accuracy on a held-out suite of 70 realistic multi-tool queries, against a roughly 21% naive keyword-search baseline on the same suite. The baseline matters more than the headline, because it is the difference between "the pipeline works" and "an embedding lookup would have done fine." The grader is deliberately unforgiving. A case passes only if every required tool appears in the result and no forbidden tool appears, where forbidden: ["*"] means the correct answer is no tools at all. For allow_any_of groups, which are sets of interchangeable tools, exactly one member must be present. Returning two valid alternatives is a failure rather than a hedge, because it pushes the choice back onto the model we are trying to protect. When an order is specified, the returned tools have to contain it as a subsequence, with relative order preserved and unrelated tools allowed to interleave. And when a case is marked should_clarify, the router must return zero tools and set the clarify flag, which alone decides the case. That last rule is the one I would defend hardest. An accuracy metric that does not reward asking instead of guessing quietly incentivizes overconfident wrong answers, because the model learns that any answer beats admitting ambiguity. Ours treats correct abstention as a pass, which means the 88% includes the system knowing what it does not know. One honest limitation: cases requiring the same tool to be called multiple times are structurally unsatisfiable, because route() emits each tool id once. We count those as real misses rather than excluding them, so the reported number is a floor rather than a flattered figure. The decisions that mattered, and the alternatives we rejected Most of the interesting engineering here is not in what the pipeline does. It is in the four or five places where we deliberately chose the harder option. Dynamic-K over fixed top-k The obvious design is "return the top 5 matches." We rejected it because a fixed k is wrong in both directions at once. For a query that needs one tool, top-5 injects four irrelevant schemas and reintroduces exactly the selection-confusion problem the router exists to solve. For a genuine multi-step request, top-5 might truncate a plan that needed six. And for an out-of-scope question, top-5 confidently returns five wrong tools. Making k dynamic means the judge has to answer "how many?" as well as "which?", which is a harder prompt and a harder thing to evaluate. It was still the right trade. Our measured average is 1.2 tools per call out of 142, which no fixed k would have produced. One hard-coded rule, and only one There is a real temptation to encode catalog-specific heuristics, something like "queries mentioning 'log' should prefer the observability server." We kept exactly one code-side rule: an absolute cosine floor for out-of-scope detection. The reasoning is a division of labor. That floor is the one judgment the LLM cannot make cheaply. To know that nothing in the catalog fits, a judge would have to see the entire catalog, which is precisely the cost we are eliminating. A cosine threshold answers it in one vector op. Everything else (which tools, dedup, ambiguity, ordering) is semantic work and gets delegated to the judge. Rules that encode catalog specifics would need rewriting every time someone connects a new backend, which defeats the pluggability goal. Judge order, not score order Early on we sorted the returned tools by cosine score. That is wrong for multi-step requests. "Pull last week's errors and open a ticket for the worst one" has an inherent execution order that has nothing to do with which tool embeds closer to the query. The judge reasons about sequence, so the judge's output order is the router's output order, and we explicitly do not re-sort. Filter unhealthy backends before the judge, not after The intuitive design for backend health is retry-on-failure: select a tool, call it, and handle the error if the backend is down. We invert it. A backend marked down has its tools excluded from the candidate set before the judge sees them, re-evaluated on every call. This matters because of the specific failure it prevents. The judge picks the perfect tool, explains its reasoning, and then invoke fails, so the user gets a wrong-looking answer for a right-looking decision. Filtering early means the judge selects the best reachable tool instead, possibly a second-choice tool on a healthy backend, which is the correct behavior. We also chose reactive health detection over a heartbeat loop. A backend gets marked down on an actual failed call and retried on next use. A separate polling process is one more thing to keep in sync with reality, and health checks that themselves flake produce false "down" states. Deprecated tools stay fetchable Our first filter dropped every tool marked deprecated. We removed that clause deliberately, because a request like "export it in the legacy format" specifically needs the deprecated version. Governance filtering still applies, with PII-touching tools gated behind an explicit flag and destructive tools behind another, but deprecation is metadata for the judge to weigh rather than a hard exclusion for code to enforce. Keeping the registry honest A tool router is only as good as its index, and tool catalogs are not static. Backends add tools, deprecate others, and change descriptions. Switchboard runs a background ingestion pipeline, independent of request-time traffic, which polls every registered backend on an interval and also reacts immediately to admin-triggered registration or removal. Same pipeline, two triggers. Each tool runs through enrichment: deterministic structural and length checks, LLM-based description-quality and off-topic detection, TF-IDF duplicate and outlier detection, and incremental LLM clustering against a persisted taxonomy. There is a circuit breaker on the LLM enrichment step. If fewer than half the tools survive validation, the pass is treated as an outage and the registry is left untouched. This was a deliberate answer to a real failure mode, since a flaky upstream model silently emptying your tool catalog is worse than having no enrichment at all. Before touching Pinecone the pipeline diffs against a Redis hash cache, so a full re-poll of a backend only costs an embedding call for what actually changed rather than the whole catalog every time. Each tool is then embedded as three views (name, description, parameters) into dense and sparse Pinecone indexes. That three-view embedding deserves a note. A single embedding of a concatenated tool blob dilutes each signal, and a distinctive parameter name gets averaged into prose. Embedding name, description, and parameters separately, then collapsing to the best-scoring view per tool at query time, means a query that matches strongly on one dimension still surfaces the tool instead of being averaged into mediocrity. Plugging in a new backend without touching a running system This was a hard requirement from day one. Adding or removing a backend MCP server should never mean a restart, a redeploy, or a config-file hand-edit under load. curl -X POST http://:/admin/backends \ -H "X-Admin-Token: $ROUTER_ADMIN_TOKEN" -H "Content-Type: application/json" \ -d '{"server_id":"firecrawl","transport":"http", "url":"https://my-mcp-server.example.com/mcp", "auth_header":"Authorization","auth_value":"Bearer "}' One authenticated call hot-connects the backend. Every tool it exposes is validated and embedded through the same enrichment pipeline the interval process uses, backends.json is updated, and the moment the call returns those tools are live and discoverable through find_tools. No restart, no downtime for existing traffic. Removal is the mirror image: one DELETE call clears Pinecone, Redis, and the config in one step, so you never leave orphaned vectors behind. Registration is serialized behind a lock so two concurrent calls cannot race on shared state. It is an unglamorous detail that matters the first time two people onboard backends simultaneously. On the host side, plugging in is equally uneventful. Switchboard speaks standard MCP over HTTP, so it is just another entry in mcp.json. We wired it into Claude Code with zero special prompting, no "please use the router" instruction needed. A plain question like "show me the error logs for checkout-api in the last hour" triggers find_tools automatically, gets back exactly the one tool that matters, and resolves. That was the actual bar we were aiming for. The router should be invisible when it is working. What this actually saves We instrumented every routing decision end to end. Every find_tools call writes a telemetry row recording the routed-versus-full-catalog token estimate, search latency, judge latency, and outcome. Those writes are fire-and-forget, so they never block the response even if the DB write fails. A Streamlit dashboard reads that table live, and nothing on it is hardcoded. Measured against the 142-tool catalog, routing cuts input tokens per call by about 99.6% versus sending the full catalog every time, which works out to roughly 28,700 tokens saved per call (about 124 tokens routed against about 28,860 for the full catalog dump). The average call selects 1.2 tools out of 142 available, which is the dynamic-k judge working as intended rather than a fixed top-k masquerading as precision. End-to-end latency averages about 6.5 seconds, with roughly 4.9 seconds of that sitting in the LLM judge step. That is the honest cost of doing selection properly instead of guessing. The clarify rate is about 12%, meaning genuinely ambiguous requests get kicked back to the user instead of the router guessing and invoking the wrong tool. Projected from those per-call savings, at 1,000 calls a day (one active team's traffic) and Sonnet-class input pricing, that comes to roughly 10.5 billion tokens and about $31,000 a year in avoided token spend. The per-call saving is measured. The annual figure is that measurement multiplied by an assumed call volume, and it scales linearly with whatever volume your deployment actually sees. Why a tool router matters more than it looks like it should It is tempting to file this under prompt optimization. It isn't. As agentic systems connect to more of an organization's real infrastructure, three things become true at once. Tool catalogs only grow, because nobody removes MCP servers once they are useful, they just add more. Context windows are not free even when they are technically large enough, since every token spent on tool schemas is a token not spent on reasoning, retrieved context, or conversation history. And selection errors compound, because a wrong tool call against a real backend (sending an email, querying production data, triggering a workflow) is not a cosmetic bug. It is an action with consequences. A router answers all three at once. It decouples how many tools exist from how many tools the model has to reason about, and it puts a validating, explainable decision layer between "the user asked for something" and "a real backend got called." Handling the edges A few decisions exist specifically because the happy path is not where systems like this actually fail. If the judge is unavailable or its call fails, routing falls back to dense top-5 rather than returning nothing, because a degraded answer beats an outage. The admin API 404s entirely unless an admin token is configured, so there is no insecure-by-default state to accidentally ship. When find_tools cannot confidently resolve intent the system asks rather than picking the closest-sounding tool and hoping. Instrumentation writes are fire-and-forget, so a Postgres hiccup degrades your dashboard and never your response latency. And ingestion is idempotent and incremental, so backends can be re-polled constantly without re-embedding a catalog that has not changed. What's next The current bottleneck is not accuracy. It is the 4.9-second judge step, plus the fact that the system learns nothing from its own traffic. Both are addressable. Draining admin-triggered ingestion asynchronously. Registering a new backend currently blocks the API call until every tool is validated and embedded. That is honest but not ideal at scale. The target is to enqueue the job and let the background worker drain it so registration returns immediately. The work is already queue-shaped; the admin path just drains its own jobs synchronously today. Cutting judge latency with a cascade. Roughly 75% of end-to-end latency is one LLM call, and most queries are not close calls. When the top candidate leads by a wide cosine margin and no near-twins exist, a small fast model or even a deterministic path could resolve it, escalating to the full judge only for genuinely contested cases. The measurement to run first is what fraction of traffic is actually contested. If it turns out to be 20%, a cascade cuts average latency substantially at close to zero accuracy cost. Query rewriting before retrieval. Terse or jargon-heavy requests like "502s on checkout" embed poorly against prose tool descriptions. Expanding the query before it hits Pinecone closes the vocabulary gap between how users ask and how tools are documented. The trade-off is another model call in the hot path, which argues for doing it only when the first retrieval scores weakly. Rewrite as a fallback, not a default. Hard negative mining from real clarify and miss cases. Every clarify is a labeled example of "these tools looked similar but the query was ambiguous," and every user retry after a bad selection is a labeled negative. Mining those into a fine-tune or a re-ranker trains on our own traffic distribution instead of generic semantic similarity. This is the highest-leverage accuracy lever we have, because it compounds. More traffic means better signal. Feedback-loop learning from invoke outcomes. We currently log which tools were selected, but not whether the invocation succeeded or whether the user immediately tried something else. Closing that loop, from selection through invocation to outcome, turns the telemetry table from a cost dashboard into a training set. It is mostly a schema change and a follow-up write. Per-tenant re-ranking. Usage patterns differ sharply by deployment. A team that lives in observability tooling should see different ranking than one that lives in CRM tools. A lightweight per-tenant prior over the global model captures most of that gain without maintaining separate indexes. Tightening the enrichment quality bar. Bad retrieval is often a symptom of bad tool descriptions rather than a bad retriever. Flagging ambiguous or poorly-described tools at ingestion time, and reporting them back to the backend owner, fixes the problem at its source instead of compensating for it at query time. Cross-system observability. The dashboard reports router cost, latency, and outcome. Unifying it with orchestrator-side telemetry, whether OTel/SigNoz/Langfuse or a shared events table, would let cost and accuracy be viewed end to end rather than only at the router boundary. The takeaway The interesting problem in agentic AI right now is not whether the model can call a tool. It is whether the model can call the right tool, reliably, as the number of tools you connect grows past what any single prompt should ever have to describe. That is an infrastructure problem rather than a prompting problem, and infrastructure problems deserve infrastructure solutions: a registry, a retrieval pipeline, a validation layer, and telemetry that tells you the truth about whether it is working. That is Switchboard. One operator, any number of lines, and a host that never has to know the number.