Recompute Your Context Budget Before You Move an AI Workflow to a Free Model Tier

Recompute Your Context Budget Before You Move an AI Workflow to a Free Model Tier The quietest failure in an AI tool migration is not the endpoint going dark; it is your prompt builder silently dropping a different set of files once the token accounting changes underneath it. You can repoint the client at a new base URL, watch the first response stream back correctly, and still ship a review agent that reads the wrong half of every pull request. Before you treat free model access as a drop-in replacement, recompute the input budget and prove which files your truncation rule now discards. Everything below is a hypothetical walkthrough, not a report of a migration anyone completed. The scenario is deliberately ordinary: a code review assistant pinned to a paid hosted model, with a prompt builder that assembles a system prompt, a few tool schemas, and the changed files under a character-based limit. You move that assistant to a free tier, the request path is byte-for-byte identical, and yet reviews on larger diffs start missing context that used to be present. Nothing crashed, which is exactly why this class of regression survives a cutover unnoticed. 1. Why the arithmetic breaks when the code does not Three numbers define the space you actually have: the context window of the target model, the output tokens you reserve, and the cost of your own text measured in that model's tokenizer. A character limit is only a proxy for the third number, and it is calibrated to one tokenizer on one day. The moment the tokenizer changes, your proxy keeps producing a number while losing its connection to reality, and the truncation rule that depended on it starts keeping a different amount of code. The second leftover is the reserved output budget. Most teams copy max_tokens from the old config into the new client and never revisit it, so a value tuned for one model quietly over-reserves or under-reserves on another. The third leftover is behavioural: if overflow is handled by slicing the first N characters, your builder is biased toward the top of the diff and will never tell you what it skipped. None of these problems require a new feature to fix. They require you to stop converting tokens with a ratio and start measuring them with the tokenizer that the target model actually ships. 2. Step 1 — Inventory every place you encoded the old vendor Start by finding the assumptions, not by rewriting the client. Search for character heuristics, magic token constants, and truncation calls across the request path, including configuration files that hold limits outside the code. # Illustrative only; not executed against a real repository. rg -n --hidden -g '!node_modules' -g '!*.lock' \ -e 'max_tokens' -e 'MAX_TOKENS' -e 'context_window' -e 'CHARS_PER_TOKEN' \ -e 'char_limit' -e 'truncate' -e 'slice\(0,' -e 'len\([^)]*\)\s*/\s*4' \ . Treat every hit as a decision you have to re-make rather than a line you have to change. A len(text) / 4 conversion in a helper is a tokenizer assumption in disguise, and a slice(0, N) in the assembler is a policy about which context is disposable. Write down the file, the assumed limit, and who owns it, because the gate in step 4 will ask you to close each row. 3. Step 2 — Replace the proxy with a measured budget ledger The replacement for a character heuristic is a small function that subtracts fixed costs first and then admits files in priority order until the budget is exhausted. Keep the tokenizer behind an interface so you can swap it when the target model changes, and make every rejection observable rather than silent. # Illustrative sketch. Not executed; the tokenizer call is a placeholder. from dataclasses import dataclass @dataclass(frozen=True) class Window: total: int # context window reported for the target model reserved_output: int reserved_tools: int # system prompt + tool schemas safety: float = 0.05 @property def input_budget(self) -> int: spendable = self.total - self.reserved_output - self.reserved_tools return int(spendable * (1 - self.safety)) def count_tokens(text: str, model: str) -> int: raise NotImplementedError("wire in the tokenizer shipped with your target model") def assemble(system: str, tools: str, files: list, window: Window, model: str) -> list: budget = window.input_budget used = count_tokens(system, model) + count_tokens(tools, model) kept, dropped = [], [] for f in files: # caller supplies files in priority order cost = count_tokens(f.text, model) if used + cost