Tokenization Up Close

I What Tokenization Actually Is

When you type a prompt, you see words. The model sees something very different: a sequence of integers.

Tokenization is the process of splitting your text into tokens — subword units from a fixed vocabulary — and mapping each one to an integer ID. The model never touches your raw text. It receives a list of numbers like [791, 2027, 6743, 374, 2294]and generates another list of numbers in return. Everything between "text in" and "text out" is integer math.

Consider the word "unhappiness." You see one word. The tokenizer might split it into ["un", "happi", "ness"]— three tokens, three integers. The word "the" is common enough to be a single token. The word "counterintuitively" might be four or five. There's no clean rule. Every split depends on the tokenizer's learned vocabulary.

II Why Subwords, Not Words

Tokenizers could split text into whole words. Or individual characters. They do neither, and the reason matters.

A word-level vocabularywould need millions of entries — every word in every language, every conjugation, every proper noun. "Running," "runs," "runner," and "ran" would each need a separate entry. New words (brand names, slang) would be out-of-vocabulary errors. It doesn't scale.

A character-level vocabularywould be tiny — maybe 256 entries. But individual letters carry almost no meaning. The model would need to learn that "c-a-t" means the same thing as the concept of a cat, spending enormous capacity on spelling instead of semantics. Sequences get impossibly long.

Subword tokenization(BPE, SentencePiece, WordPiece) is the sweet spot. The algorithm starts with individual characters and iteratively merges the most frequent pairs until it reaches a target vocabulary size — typically 30,000 to 100,000 entries. Common words like "the" become single tokens. Rare words get split into recognizable pieces. Every possible string can be encoded, even words the model has never seen.

This is why a vocabulary of 100K tokens can cover every language on Earth. English common words get single tokens. Rare English words get split into 2-3 pieces. Non-English text gets split into more pieces — which has real cost implications we'll get to shortly.

See how it works on real text:

III The Mysteries Tokenization Explains

Once you understand tokenization, a whole category of "weird AI behavior" suddenly makes sense.

Why can't the model count the letters in "strawberry"? Because it never sees letters. "Strawberry" might be tokenized as ["str", "aw", "berry"] or ["straw", "berry"]. The model receives two or three token IDs. It has no direct representation of the individual characters s-t-r-a-w-b-e-r-r-y. Asking it to count "r"s is like asking someone to count the bricks in a photo of a house — the information was lost in the representation.

Why is code more expensive to process than English? Code uses lots of special characters, indentation, and uncommon symbol combinations. response.data.map((item) =>gets split into far more tokens than "the cat sat on the mat" despite being roughly the same length. More tokens means more cost and more context window consumed.

Why do non-English languages cost more? English dominated the training data that built most tokenizer vocabularies. English words got more merges, more dedicated tokens. A Chinese sentence or Hindi paragraph expressing the same idea may require 2-3x as many tokens. Same meaning, higher bill.

Why does "3+7" sometimes produce wrong answers?Numbers are tokenized unpredictably. "127" might be one token, or it might be split into "12" and "7" or "1", "27". The model has no concept of place value — it's doing pattern matching on token sequences, not arithmetic. Depending on how the numbers split, the patterns it learned during training may or may not apply.

IV Token Economics

Every API call has two token counts that determine your bill: input tokens (your prompt, system message, and any context) and output tokens(the model's response). Output tokens typically cost 2-5x more than input tokens because they require more computation to generate.

The widely cited ratio — 1 token is approximately 0.75 words(or 1 token is roughly 4 characters) — is a useful estimate for English prose. But it's wildly wrong in specific cases:

• Code:1.5-2.5x more tokens per "word" than English prose. Curly braces, operators, and indentation all consume tokens.
• JSON: The most token-hungry format. The structural overhead (keys, colons, braces, quotes) can double or triple the token count versus the same data in plain text.
• Non-English: Chinese, Japanese, Korean, Hindi, Arabic — all require more tokens per concept than English. A Japanese product serving Japanese users pays a token tax on every request.
• System prompts: Your system prompt is sent with every single request. A 500-token system prompt at 10,000 requests/day is 5 million tokens/day just in system prompt overhead. That adds up fast.

And here's a fact that surprises most builders: the same text produces different token counts across different models. Claude, GPT, Gemini, and Llama all use different tokenizers with different vocabularies. A prompt optimized for one model's tokenizer may cost more on another. See it for yourself:

V Practical Implications for Builders

Always count tokens, never estimate. Every major provider offers a tokenizer library: OpenAI has tiktoken, Anthropic publishes token counts in API responses, and open-source models ship their tokenizers. Use them. An estimate of "about 1,000 tokens" might be 800 or 1,400 in practice, and that gap compounds across millions of requests.

Prompt compression is a real optimization lever. Restructuring text to use fewer tokens without losing meaning is free money. Consider the difference:

Before:"Please carefully analyze the following text and provide a comprehensive, detailed summary that captures all of the key points and important details."

After:"Summarize this text, capturing all key points."

The second version says the same thing in roughly half the tokens. Multiply that savings across every request and it's a meaningful cost reduction.

Token-aware truncation matters. When you need to fit text into a context window, naive character truncation (cutting at 10,000 characters) can split a token in half, creating garbage input. Always truncate on token boundaries. Better yet, truncate on semantic boundaries — sentence or paragraph breaks — while counting tokens to stay within limits.

VI The Tokenizer as Product Constraint

Here's the part most builders miss: the tokenizer is baked into the model at training time.You can't change it. You can't swap in a different one. The entire model — all its learned patterns, all its billions of parameters — is wired to one specific tokenizer vocabulary.

This has several implications that matter for product decisions:

Different models tokenize differently. The same text might be 100 tokens on Claude and 120 tokens on GPT. That 20% difference flows directly to cost and context window utilization. When comparing model costs, comparing price-per-token is only half the story — you need price-per-equivalent-output.

Tokenizer quality varies by language. If your product serves a multilingual audience, tokenizer efficiency in non-English languages becomes a real variable. A model with a tokenizer trained on more diverse data will be cheaper to run for non-English users. This is a hidden factor in vendor selection that rarely shows up in benchmark comparisons.

Tokenizer evolution is slow.Unlike model weights, which get updated with each new version, tokenizers change rarely. When they do change (as happened when GPT-4 moved from GPT-3's tokenizer), it can break token-counting code, invalidate cached token counts, and shift cost estimates. Plan for it.