Your LLM Context Window Is Lying to You: How Token Budgets Actually Work

The context window number in your model's docs is a capacity spec, not a promise. A model that advertises 200K tokens will happily accept 200K tokens, but the quality of what it does with them starts sliding long before you reach that ceiling. If you build production features assuming the whole window works equally well, you will ship bugs that only show up on long inputs. Here is how token budgets actually behave, and how to stop the window from quietly lying to you. The number on the box is not the number you get Advertised context windows describe capacity, not usable quality. Work on long context behavior shows models paying far more attention to the beginning and end of the input, while content buried in the middle gets missed even when you stay well inside the stated limit. People call this the lost in the middle problem, and it comes from how attention works, not from a bug you can prompt your way around. The gap is wider than most teams expect. Models claiming a 200K window show measurable quality degradation around 130K tokens in practice. That is not the model refusing to answer. It is the model getting quietly worse at using the tokens you paid to send. If a critical instruction sits in the middle of a huge prompt, treat it as maybe read, not definitely read. Everything shares one budget The biggest misconception is that the context window is only for your input. It is not. The limit applies to the total of input and output tokens combined. Your system prompt, the conversation history, any retrieved documents, the user query, and the model's own response all draw from the same pool. That has a consequence people hit constantly. A generous system prompt plus a long chat history can leave almost no room for the answer. The model does not warn you first. It runs out of budget mid thought and the response gets truncated, or the API rejects the request outright. Once you accept that output competes with input for the same space, you stop being surprised by cut off answers on your longest sessions. Why big contexts get slow and expensive Long prompts do not just risk quality. They cost you time and money on every call. The attention step compares every token to every other token, so the core computation grows with the square of the input length. The QK^T matrix is n by n, which means doubling your context roughly quadruples the work the model has to do. One study measured a 7x latency increase at 15,000 words of context. That is the difference between a snappy reply and a spinner your users abandon. Cost follows the same curve, because LLM APIs charge per token for both input and output. Every extra token of history or retrieved context is money you spend on every single request, whether or not it earned its place. If your bills keep climbing, oversized prompts are usually part of the story, and trimming them is one of the fastest ways to reduce inference costs without changing your model. Count tokens before you send them You cannot manage a budget you never measure. Before firing a request, count what each part of the prompt actually costs. This one habit surfaces the system prompt bloat and runaway history that silently eat your window. import { encoding_for_model } from "tiktoken"; const enc = encoding_for_model("gpt-4o"); const count = (text) => enc.encode(text).length; const parts = { system: count(systemPrompt), history: count(conversation.map((m) => m.content).join("\n")), docs: count(retrievedChunks.join("\n\n")), query: count(userQuery), }; const inputTokens = Object.values(parts).reduce((a, b) => a + b, 0); console.log(parts, "input total:", inputTokens); Run this once against a real session and you will usually find one part hogging the budget. Nine times out of ten it is either a bloated system prompt nobody has trimmed in months, or an unbounded history that grows every turn. Watch the budget in production Local counting is step one. In production you want the budget checked on the hot path and an alert before you hit the wall, not after. The rule that works: log token usage on every call, and fire an alert when usage crosses 80 percent of the context limit. That gives you room to react before requests start failing. const CONTEXT_LIMIT = 128000; // set this to your model's real limit function checkBudget({ inputTokens, maxOutputTokens }, requestId) { const projected = inputTokens + maxOutputTokens; const usage = projected / CONTEXT_LIMIT; console.log(JSON.stringify({ requestId, inputTokens, maxOutputTokens, usagePct: Math.round(usage * 100), })); if (usage > 0.8) { notifyOncall(`Context at ${Math.round(usage * 100)}% on ${requestId}`); } return usage