BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. When you type: unbelievableness an LLM does not see the word. It sees something more like: ["un", "believ", "ableness"] Or perhaps: ["un", "believe", "ness"] Or, depending on the tokenizer: ["un", "bel", "iev", "ab", "leness"] That difference is not cosmetic. Tokenization determines the length of the model's input sequence, which affects context usage, inference cost, attention computation, vocabulary size, handling of rare words, programming-language behavior, multilingual performance, and even some model failure modes. And one of the most widely used ideas behind modern LLM tokenizers has an unusually non-LLM origin: a 1994 data-compression algorithm by a programmer named Philip Gage. The basic idea is remarkably simple: Find things that occur together often, and give them a reusable symbol. That idea eventually went from C programmers doing data compression, to neural machine translation, to GPT-2 and the tokenization machinery surrounding today's language models. This article builds the idea from intuition to implementation, then looks at the less obvious engineering consequences. 1. What problem is a tokenizer actually solving? A neural network wants numbers. Your input is text: The server returned HTTP 500. The model needs: [the, server, returned, HTTP, 500, .] which eventually becomes integer IDs such as: [464, 2126, 4710, ...] The obvious question is: Why not make every word a token? Suppose the vocabulary contains: cat dog server database running ... Now consider: microarchitectural microarchitectures microarchitecturally You immediately run into the open-vocabulary problem. There are infinitely many possible strings. New product names appear. Developers invent identifiers. People misspell things. Languages generate long compounds. Users paste URLs, hashes, code, emojis and arbitrary Unicode. A word-level tokenizer therefore needs some fallback mechanism. At the other extreme, we could tokenize one character at a time: m i c r o a r c h i t e c t u r a l Now everything is representable, but sequences become much longer. That creates a fundamental tradeoff: word tokens tokenizer -> token count -> context utilization -> compute + memory + latency Token efficiency is also model capacity Imagine two representations of the same sentence: Tokenizer A: 12 tokens Tokenizer B: 18 tokens The model using B has to predict a longer sequence. At training time that means more prediction positions. At inference time it means more autoregressive steps. For APIs, token count also becomes a billing and capacity unit because providers commonly meter usage in tokens. So tokenizer quality is not merely: "Does the text tokenize?" It is also: "How economically does this representation use the model's finite sequence budget?" Code exposes the problem Consider: def calculate_monthly_revenue(customer_transactions): ... A tokenizer that is optimized around English prose may discover useful units such as: calculate monthly revenue customer But source code contains many patterns that have different frequency distributions: __init__ HTTPRequest std::unordered_map get_user_profile ===> Programming languages are therefore an interesting tokenizer workload because identifiers, punctuation, whitespace, delimiters and repeated syntactic fragments all compete for vocabulary capacity. The result is one reason why "tokenizer efficiency" should be evaluated on the actual distribution your model serves, not only on generic English text. 7. At inference time, BPE is a deterministic compression dictionary Once training is finished, the tokenizer no longer needs to "discover" anything. It has two important artifacts: vocabulary merge rules For example, imagine the merge ranking contains: 1. e s 2. es t 3. n e 4. ne w 5. new est ... Now given: newest the encoder applies the learned rules in their defined priority. Conceptually: n e w e s t then perhaps: ne w e s t then: ne w est then eventually: new est depending on the learned merge table. The output is something like: [new, est] The exact implementation used by modern tokenizers is optimized considerably beyond this toy procedure. A naive implementation that rescans an entire corpus after every merge would be unnecessarily expensive. But the conceptual model remains: base symbols + ordered merge rules = tokenizer And that has a subtle consequence for developers: token IDs are meaningless without the tokenizer definition that produced them. Token ID: 12345 does not inherently mean "hello" or "database." It means whatever entry 12345 refers to in a particular tokenizer vocabulary. This is also why changing tokenizers can invalidate embeddings, model inputs, cached token sequences and various pieces of preprocessing infrastructure. The tokenizer is effectively part of the model's interface contract. 8. What BPE does not solve BPE solves one problem very well: How do we turn arbitrary text into a finite vocabulary while giving common sequences compact representations? It does not solve everything. It does not guarantee linguistically meaningful boundaries. It does not guarantee equal token efficiency across languages. It does not make arithmetic easy. It does not make code identifiers naturally interpretable. It does not prevent pathological tokenizations. And it certainly does not give the model a semantic understanding of the pieces. You can see this clearly with a made-up identifier: calculateUserMonthlyNetRevenueExcludingRefunds The tokenizer might produce something like: calculate User Monthly Net Revenue Excluding Refund s Or something considerably less intuitive. That is perfectly fine from the tokenizer's perspective. Its job is not to discover what the identifier "means." Its job is to produce a sequence that fits within the vocabulary and represents the input efficiently according to patterns learned from its corpus. This also explains an important phenomenon when working with LLM APIs: two strings that humans consider almost identical can have materially different token counts. For example: camelCaseIdentifier and: snake_case_identifier may produce different segmentations because their character sequences and punctuation patterns have different statistics. Likewise: hello world and: hello_world are linguistically related but are not equivalent objects to a frequency-based tokenizer. The model ultimately sees the tokens, not our intuitive notion of "the same phrase." Conclusion: The tokenizer is the first compression algorithm in your LLM stack There is a useful way to think about the whole system. Your original text contains enormous redundancy. BPE performs a kind of learned compression: raw bytes | v frequent local patterns | v reusable subword tokens | v shorter sequence | v Transformer The irony is that the algorithm is not particularly sophisticated. Count adjacent pairs. Merge the frequent ones. Repeat. Yet that small mechanism sits directly in front of billions of neural-network parameters. And its decisions propagate everywhere: tokenizer -> sequence length -> context capacity -> attention computation -> inference latency -> memory usage -> training efficiency -> API cost -> multilingual behavior -> code handling That makes tokenization one of those pieces of infrastructure that is easy to ignore precisely because it works so well. The most interesting lesson may be historical. Philip Gage was trying to compress bytes in 1994. Sennrich, Haddow and Birch were trying to solve rare-word problems in neural translation in 2016. GPT-2 then adapted the idea to byte-level language modeling. A concept that began as a compact data-compression trick became part of the interface between human language and modern neural networks. That is a useful reminder for developers building ML systems: sometimes the important abstraction is not the complicated algorithm in the middle, but the small transformation that determines what the algorithm gets to see. What tokenization behavior have you found most counterintuitive in an LLM—code, multilingual text, numbers, punctuation, or something else? Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: HexmosTech / LiveReview Blast-Radius Aware AI Code Review for Business-Critical Systems       LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. blast-radius-demo.mp4 LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score How does Blast Radius scoring work? (a more technical explanation) Here's the goal: A 3-line fix in a function used by 40 other files, that also writes to a database, should score high. A 300-line UI change in one file, fully covered by… View on GitHub Click below to try LiveReview with your codebase: