Chapter 2 · Section 2.6 · 70 min

Tokenizer pitfalls

In his tokenizer lecture, Andrej Karpathy puts up a slide of strange things language models do. They can't spell words. They can't reverse a string. They are oddly bad at simple arithmetic. They are worse at non-English languages. They make a mess of Python indentation. They halt or babble when they see the string "SolidGoldMagikarp". They warn you about trailing whitespace. His punchline is that every one of these traces back to tokenization.

This section takes that list seriously. For each failure, we'll find the mechanism, measure it where we can, and look at the fixes people actually use. These are the questions interviewers ask to separate people who have used LLMs from people who understand them, and they are also the bugs you will hit in production.

Arithmetic and digit splitting

Here is how two tokenizers cut the same numbers (counted with tiktoken). Vertical bars separate tokens.

text	GPT-2	GPT-4 (cl100k_base)
1234567	123 \| 45 \| 67	123 \| 456 \| 7
2024	20 \| 24	202 \| 4
3.14159	3 \| . \| 14 \| 159	3 \| . \| 141 \| 59
12345 + 67890 = 80235	123 \| 45 \| ␣+ \| ␣6 \| 78 \| 90 \| ␣= \| ␣80 \| 235	123 \| 45 \| ␣+ \| ␣ \| 678 \| 90 \| ␣= \| ␣ \| 802 \| 35

GPT-2's split is whatever its merges happened to learn, so the same digits are chunked differently depending on what surrounds them, and a number's place values land in unpredictable tokens. GPT-4's pre-tokenizer (§2.3) cuts digits into runs of at most three, left to right, which makes the chunks predictable, and every string of one to three digits, all 1,110 of them, is a single token. Llama 3 uses the same rule. Llama 1 and 2, PaLM, Mistral, and Qwen2.5 split every number into single digits instead.

Why should the direction matter? Think about how you add. You line the numbers up on the right, add the ones, carry, add the tens, and so on. Left-to-right chunking aligns chunks on the left, so in 12345 + 678, the chunk 45 holds the tens and ones of the first number while 678 holds the hundreds, tens, and ones of the second. The model has to learn arithmetic on chunks whose place values do not line up, and the misalignment changes with every length. Right-to-left chunking (12 | 345 and 678) lines chunks up by place value, like a person writing digits in groups of three.

Singh and Strouse (2024) tested exactly this. They forced right-to-left chunking in GPT-3.5 and GPT-4 by writing numbers with commas every three digits from the right, and found addition accuracy up to 20 percentage points higher than with the default left-to-right chunks. The errors under left-to-right chunking were strikingly stereotyped, and concentrated where the answer has more digits than the addends, which is exactly when the misalignment between operands and answer shifts. Asking the model to first rewrite the numbers with commas recovered most of the lost accuracy.

Python · cell 1

With right-to-left chunks every chunk ends at a power of ten that is a multiple of 3 for all three numbers, so chunk $i$ of each operand always talks about the same digits. With left-to-right chunks the endings depend on each number's length.

Core

Chunk digits in either direction

Implement chunk_digits(s, k=3, direction="l2r") that splits a string of digits into chunks of at most k, either from the left ("l2r", like GPT-4 and Llama 3) or from the right ("r2l", what commas enforce). Then implement aligned(a, b, k=3), which says whether the left-to-right chunks of two numbers line up by place value. Measure every boundary between two chunks by its distance from the right end of its number, and return True exactly when the two numbers have the same set of boundary distances smaller than the length of the shorter number.

def chunk_digits(s, k=3, direction="l2r"):
    raise NotImplementedError

def aligned(a, b, k=3):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Spelling and character-level tasks

Ask a model to count the r's in "strawberry", to reverse a word, or to list the letters of a name, and you are asking about something it cannot directly see. In GPT-4's tokenizer, " strawberry" (with its leading space) is one token, id 73700. Without the space, "strawberry" is three tokens, str, aw, berry. Neither form gives the model letters. To answer, the model must have memorized, somewhere in its weights, that token 73700 is spelled s-t-r-a-w-b-e-r-r-y, and then count over that memory. Models do learn a lot about the spelling of their tokens from context (typos, rhymes, acrostics, and lists of letters all leak it), but the knowledge is indirect and patchy.

Karpathy's favorite example is .DefaultCellStyle, which is a single token in cl100k_base (it comes from a lot of C# code). Asking GPT-4 to reverse that string is asking it to reverse something it sees as one indivisible symbol.

Three workarounds follow from the mechanism.

Make the characters visible. Ask the model to first write the word with separators ("s, t, r, a, w, ..."). Each letter then becomes its own token, and counting or reversing over visible tokens is easy. This is chain of thought (§18.1) used to change the input representation.
Use a tool. Let the model call code for character-level work. It is the reliable answer.
Change the tokenizer. Byte-level models (§2.7) see every character, and ByT5's authors found them better on tasks that depend on spelling and pronunciation.

Glitch tokens and under-trained tokens

In early 2023, Jessica Rumbelow and Matthew Watkins were clustering the token embeddings of GPT-J and GPT-2 (which share a tokenizer), and kept finding the same strange tokens nearest to the cluster centers. They turned out to be the tokens closest to the centroid of all 50,257 embeddings, a list that included SolidGoldMagikarp (id 43453), TheNitromeFan (42090), davidjl (23282), RandomRedditorWithNo, and cloneembedreportprint. Prompting the models with them produced bizarre behavior. Asked to repeat " SolidGoldMagikarp", one GPT-3 model answered "distribute". Others produced evasions, insults, or unrelated words, and some broke determinism at temperature 0. Commenters soon noticed that several were Reddit usernames of people extremely active in r/counting, a forum where users count upward one comment at a time.

The widely accepted explanation is a mismatch between two datasets. The tokenizer was trained on one corpus, in which those usernames appeared so often that BPE gave them their own tokens. The model was trained on a different, filtered corpus in which those strings almost never appeared. So the vocabulary has rows that the model barely ever saw.

Why does a rarely seen token end up near the centroid, and why does it make the model misbehave? Look at the gradients.

An input embedding row gets a gradient only when its token appears in the input (the sparse-gradient fact from §1.2). A token that never appears keeps its initial value, or decays toward zero under weight decay.
An output row gets a gradient at every position, because the softmax pushes down the probability of every wrong token. For a token that is never correct, the update is always "lower my logit", applied along the average hidden state. All such rows drift in the same direction, so they end up close to one another and to a shared, uninformative vector.

When such a token is fed in, the model receives an input vector it has never learned to interpret, and whatever comes out is unpredictable. Land and Bartolo (2024) turned this into an automatic detector. Write $\mathbf{E}_{\text{out}} \in \R^{V \times d}$ for the output (unembedding) matrix, whose row $\mathbf{E}_{\text{out},i}$ produces the logit of token $i$ . Take a few tokens known to be unused (such as reserved special tokens or ids above the tokenizer's real vocabulary), average their output rows into a reference vector $\mathbf{u}_{\text{ref}}$ , and rank every token by its cosine distance to it,

C_i = 1 - \frac{\mathbf{E}_{\text{out},i}^\top \mathbf{u}_{\text{ref}}}{\lVert\mathbf{E}_{\text{out},i}\rVert\,\lVert\mathbf{u}_{\text{ref}}\rVert}, \qquad \mathbf{u}_{\text{ref}} = \frac{1}{|\mathcal{R}|}\sum_{j \in \mathcal{R}} \mathbf{E}_{\text{out},j} ,

where $\mathcal{R}$ is the set of reference ids. In plain words, tokens whose output rows point the same way as the known-dead rows are probably dead too. For models with untied embeddings, the norm of the input row is a second indicator. They then verify the top candidates with prompts that ask the model to repeat the token and check whether it can (a token is confirmed if the model gives it under 1% probability in a setting where it should be easy). They found under-trained tokens in essentially every model family they examined.

🔥 In PyTorchfor a machine with PyTorch

The indicator finds all 40 remaining never-seen tokens with no false positives, using nothing but the output matrix and ten known-dead ids.

Interview

Find under-trained tokens from the output matrix

Implement undertrained_candidates(E_out, ref_ids, top_k), Land and Bartolo's main indicator. Compute $\mathbf{u}_{\text{ref}}$ , the mean of the rows E_out[ref_ids], then the cosine distance of every row to it, and return the top_k token ids with the smallest distance, excluding the reference ids themselves, sorted from most to least suspicious.

import numpy as np

def undertrained_candidates(E_out, ref_ids, top_k):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Unreachable tokens

Land and Bartolo also point out a stranger category. Some tokens can never be produced by the tokenizer at all, because encoding their own string yields a different token sequence. A merge table learned by the training loop of §2.2 and used with the same pre-tokenizer cannot contain such a token, because encoding a token's string replays exactly the merges that built it in training. Unreachable tokens appear when the vocabulary and the encoder disagree. Pre-tokenization or normalization rules may split or rewrite a token's string before BPE sees it, tokens may be added or edited by hand, or the merges may come from a different implementation with other conventions. A hand-edited table shows the mechanism. With merges (a, b), (b, c), and then (a, bc), the vocabulary contains abc, yet encoding "abc" applies (a, b) first and ends at ab c. The model therefore never sees abc as input, and its row is as untrained as a glitch token's. Real vocabularies also contain partial UTF-8 tokens (§2.3) that only ever appear inside longer byte sequences. Checking every token $t$ for $\operatorname{encode}(\operatorname{decode}(t)) = [t]$ is a cheap audit to run on any tokenizer before training a model with it.

Core

Audit a merge table for unreachable tokens

Implement unreachable_tokens(merges). Every merge (a, b) creates the token a + b. Encode that token's string with the given encode_bpe (the rank-order encoder from §2.2, provided in the setup) and return, in merge order, the tokens whose encoding is not exactly [token].

def unreachable_tokens(merges):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Trailing whitespace and prompt boundaries

Tokens carry their leading space. In GPT-4's tokenizer, "Hello world" is Hello + ␣world, and the space belongs to the second token. Now end a prompt with a space, "Hello world␣". The tokenizer has no word to attach the space to, so it emits a lone ␣ token. In training text, a lone space token followed by a word almost never occurs, because the space is nearly always merged into the word that follows. The model's next-token distribution after a dangling space is therefore off-distribution and often worse. This is why the OpenAI playground used to warn that a prompt ending in a space may produce lower-quality output.

The same problem appears whenever a prompt ends in the middle of what would normally be one token. Scott Lundberg's example is a URL. In both GPT-2's and GPT-4's tokenizers, :// is a single token. If your prompt ends with href="http:, the colon is tokenized alone, and the model, which has seen http followed by :// thousands of times but almost never http + : + //, struggles to continue with //. The prompt boundary forced a token split that the training data never contained, the non-canonical tokenization of §2.3 in action.

The same mismatch appears when a pipeline tokenizes pieces separately and glues the ids together, for example a prompt and a response, or a template and a user message. Tokenization does not commute with concatenation, $\operatorname{encode}(a) + \operatorname{encode}(b) \ne \operatorname{encode}(a + b)$ in general, because the joint text may contain a token that spans the place where $a$ ends and $b$ begins. Here is a toy byte-pair encoder with a GPT-style pre-tokenizer and ten merges, enough to see it.

Python · cell 2

In the first two pairs the cut falls inside a chunk ( Paris and ://), so the separate encodings contain a lone space or a lone colon that the joint encoding never produces. In the third the cut falls on a chunk boundary and nothing changes. Real tokenizers behave the same way. So tokenize a whole sequence once and find the boundaries from character offsets, or make every boundary fall where BPE never merges, such as on a special token. Chat templates (§2.5) do the latter.

The fix for a prompt that must end mid-token is called token healing. Remove the last token of the prompt, and constrain the first generated token to start with the removed text. In the URL example, the prompt is cut back to href="http, and the first token must start with :, which allows the model to pick ://. Microsoft's Guidance library introduced it, and Hugging Face's generate supports it with token_healing=True.

Interview

Token healing

Implement token_heal(prompt_ids, vocab), where vocab maps token ids to their strings. Remove the last token of the prompt, and return (trimmed_ids, prefix, allowed), where prefix is the removed token's string and allowed is the sorted list of token ids whose strings start with prefix (the removed token itself is always among them). An empty prompt returns ([], "", sorted(vocab)).

def token_heal(prompt_ids, vocab):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

The multilingual token tax

Recall the opening of §2.1. The same question costs 7 tokens in English and 85 in Tamil under GPT-2's tokenizer. Here are measured counts for one sentence ("Where is the nearest train station?") in fourteen languages, with three generations of OpenAI tokenizers.

Three patterns stand out. First, the gap is enormous for languages whose scripts take two or three bytes per character and were rare in the tokenizer's training data. GPT-2 tokenizes Tamil at one token per byte, which means its vocabulary learned essentially no Tamil merges at all. Second, each generation narrowed the gap. GPT-4o's o200k_base brings Tamil from 85 tokens to 13 and Hindi from 56 to 11, mostly by spending its larger vocabulary on non-English text. Third, even the best tokenizer is not neutral, and the differences show up across many sentences, not just this one. Petrov et al. (2023) measured tokenization lengths on parallel text and found differences of up to 15 times between languages, and more than 4 times even for character-level and byte-level models, because some scripts need more bytes.

Where does the tax come from? BPE spends its merges on whatever is most frequent in the tokenizer's training text, so a language that is rare there gets few merges and stays close to one token per byte. The cell below trains a byte-level BPE with a fixed budget of 60 merges on corpora with different shares of Hindi, and counts the tokens of one English sentence and its Hindi translation.

Python · cell 3

Python · cell 4

With no Hindi in training, the Hindi sentence costs 170 tokens, one per byte, while the English one costs 19, almost one per word. This is GPT-2's Tamil in miniature. A 10% share halves the Hindi cost, and an even split brings the ratio to 1.5, at a price, since English now gets fewer merges and costs twice as many tokens as before. Even at a perfect balance, part of the gap remains, because the Hindi sentence is 170 bytes against 98 for the English, the byte-level floor that Petrov et al. measured. A fixed merge budget is shared among languages, which is why multilingual tokenizers need both a balanced training mix and a larger vocabulary.

Why it matters goes beyond the bill.

Cost and latency. API prices are per token and generation time grows with output length, so speakers of high-token languages pay more and wait longer for the same answer (Ahia et al. 2023 documented this across commercial APIs).
Context. A 128K-token window holds far less Tamil text than English text.
Quality. Over-segmented languages get words chopped into near-meaningless pieces, and Rust et al. (2021) found that high fertility goes with worse downstream performance.

The fixes are to train the tokenizer on a deliberately balanced multilingual corpus, to use a larger vocabulary, and to measure fertility for every language you serve, not just English.

Code and whitespace

Code is full of whitespace, and early tokenizers treated it badly. GPT-2 has no tokens for runs of spaces, so the 4-space indent in return x costs three separate space tokens (the fourth space attaches to ␣return). That line is 5 tokens in GPT-2 and 3 in GPT-4. A short function, def f(x):\n return x + 1\n, is 14 tokens in GPT-2 and 11 in GPT-4. Multiply by deeply nested code and the waste becomes a large fraction of the context. GPT-NeoX-20B (Black et al. 2022) fixed this by adding tokens for runs of 1 to 24 spaces, and GPT-4's pattern groups whitespace runs and newlines.

Formats matter too. A small JSON object, {"name": "Ada", "age": 36, "languages": ["English", "French"], "active": true}, takes 26 tokens in GPT-4's tokenizer, while the same data in YAML takes 23. Quotes, braces, and commas all cost tokens. When you pay per token or fight for context, the serialization format is part of the prompt design.

Train-serve tokenizer mismatch

The last pitfall is the most mundane and the most expensive. The model must be served with exactly the tokenizer it was trained with, and "exactly" includes more than the vocabulary file.

A different vocabulary or merge table. Ids then point at the wrong embedding rows. The model still produces fluent-looking output, which is why this bug survives code review.
Different special-token handling. A missing or doubled BOS, or a chat template that differs from training by a newline (§2.5).
Different normalization or whitespace options. NFKC in one pipeline and none in the other, or a different setting of options like adding a prefix space, so that the first word of every input is tokenized differently.
A library upgrade. Tokenizer libraries have changed defaults between versions, for example in how spaces next to special tokens are handled.

The defenses are cheap. Version and hash the tokenizer files together with the model weights. Keep a set of canary strings (mixed scripts, code, numbers, emoji, special tokens) with their expected token ids, and check them in both the training and the serving stack on every deploy. Assert the round trip $\operatorname{decode}(\operatorname{encode}(s)) = s$ on a sample of real traffic.

🎯 Interview questions on this section

Why are LLMs bad at counting the letters in a word, and how would you work around it?

The model reads tokens, not characters. In GPT-4's tokenizer " strawberry" is a single token (id 73700), so the letters are not in the input at all, and the model must rely on indirectly learned knowledge of each token's spelling. Workarounds are to ask it to spell the word with separators first (so each letter becomes a token), to have it call code, or to use a character- or byte-level model. Chain-of-thought helps precisely because it rewrites the input into a representation where the letters are visible.

How does digit tokenization affect arithmetic, and what is the best scheme?

Addition works right to left by place value. GPT-2's BPE chunks numbers inconsistently, and GPT-4's left-to-right chunks of three digits misalign place values whenever operands have different lengths, so the model must learn chunk arithmetic whose alignment changes with length. Singh and Strouse (2024) showed that forcing right-to-left chunking with commas raised GPT-3.5's addition accuracy by up to 20 points, with left-to-right errors concentrated where the answer is longer than the addends. Single-digit tokenization (Llama 1 and 2, PaLM, Mistral, Qwen2.5) avoids the problem at the cost of longer sequences, and right-to-left chunking is a good compromise.

What was SolidGoldMagikarp, and how would you find tokens like it in a new model?

A GPT-2/GPT-3 token, a Reddit username from r/counting, that existed in the tokenizer because its training data contained the name many times, but almost never appeared in the model's training data. Its embedding was essentially untrained, so prompts containing it produced bizarre outputs. To find such tokens, use the fact that never-correct tokens' output rows all receive the same push-down gradient and so align with each other. Average the output rows of known-unused tokens into a reference vector, rank all tokens by cosine distance to it (and, for untied models, by small input-embedding norm), then verify the top candidates with repetition prompts, as Land and Bartolo (2024) did. Also check reachability, $\operatorname{encode}(\operatorname{decode}(t)) = [t]$ .

Why can ending a prompt with a trailing space hurt generation quality? What is token healing?

Byte-level BPE tokenizers attach the space to the start of the next word, so in training text a lone space token followed by a word almost never occurs. A prompt ending in a space forces a lone ␣ token, putting the model off-distribution for its next prediction. The same happens whenever a prompt ends partway through a common token, such as http: when :// is one token. Token healing removes the prompt's last token and constrains the first generated token to begin with the removed text, so the model can choose the canonical token (://, or ␣world).

A product team wants to launch in Hindi and Tamil. What tokenizer questions would you ask?

What fertility and bytes per token does our tokenizer achieve on representative Hindi and Tamil text compared with English, and what does that imply for cost, latency, and how much fits in the context window? Measured on one sentence, GPT-2's tokenizer used 8 to 12 times as many tokens as English for these languages, while o200k_base is under 2 times. Does the tokenizer have enough non-English merges, or should we use a model with a larger, multilingual vocabulary? How was the model's quality evaluated in these languages, since over-segmentation correlates with worse performance? And is pricing fair to users if they pay per token?

After deploying a fine-tuned model, its outputs are fluent but subtly worse than in offline evaluation. What tokenizer-related causes would you check?

A different tokenizer version or vocabulary file in serving, a chat template that differs from training (system prompt, newlines, missing generation prompt), missing or doubled BOS, special-token strings in user content being parsed as control tokens, different normalization or prefix-space settings, and prompts that end with a space or mid-token. I would diff the token ids of canary strings and of real logged prompts between the training and serving stacks, and verify round trips.

🧭 What to remember

Arithmetic suffers when digit chunks misalign place values. Right-to-left chunks (via commas) raised GPT-3.5's addition accuracy by up to 20 points, and many models use single digits.
Spelling and character tasks fail because tokens hide letters. Making characters visible, with separators or code, fixes most of it.
Glitch tokens are vocabulary entries the model barely saw. Their output rows align with each other, which is how Land and Bartolo detect them automatically, and some tokens are unreachable by encoding at all.
Prompts that end with a space or mid-token are off-distribution, and so are sequences glued together from separately tokenized pieces, since $\operatorname{encode}(a) + \operatorname{encode}(b)$ need not equal $\operatorname{encode}(a + b)$ . Token healing backs up one token and constrains the next.
Tokenizers tax some languages many times over in cost, context, and quality, because BPE spends its merges on what its training text contains most. Code formatting and data formats change token counts too.
Next, §2.7 asks how large a vocabulary should be, and whether we need a tokenizer at all.

Rumbelow & Watkins (2023). SolidGoldMagikarp (plus, prompt generation). The discovery of glitch tokens, with the closest-to-centroid lists and the bizarre completions.
Land & Bartolo (2024). Fishing for Magikarp, Automatically Detecting Under-trained Tokens in Large Language Models. Output-embedding indicators, verification prompts, and results across many model families.
Singh & Strouse (2024). Tokenization counts, the impact of tokenization on arithmetic in frontier LLMs. Left-to-right versus right-to-left digit chunking, with a detailed error analysis.
Petrov et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. Up to fifteen-fold differences in token counts for the same content.
Ahia et al. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. The cost and quality consequences for API users.
Lundberg & Ribeiro (2023). The Art of Prompt Design, Prompt Boundaries and Token Healing. The URL example and the token healing algorithm, as a runnable notebook in the Guidance repository.
Black et al. (2022). GPT-NeoX-20B, An Open-Source Autoregressive Language Model. Section 3.2 and Appendix E on a tokenizer designed for whitespace and code.

text	GPT-2	GPT-4 (cl100k_base)
1234567	123 \| 45 \| 67	123 \| 456 \| 7
2024	20 \| 24	202 \| 4
3.14159	3 \| . \| 14 \| 159	3 \| . \| 141 \| 59
12345 + 67890 = 80235	123 \| 45 \| ␣+ \| ␣6 \| 78 \| 90 \| ␣= \| ␣80 \| 235	123 \| 45 \| ␣+ \| ␣ \| 678 \| 90 \| ␣= \| ␣ \| 802 \| 35

Zero to Frontier is made for larger screens

Tokenizer pitfalls

Arithmetic and digit splitting

Chunk digits in either direction

Spelling and character-level tasks

Glitch tokens and under-trained tokens

Find under-trained tokens from the output matrix

Unreachable tokens

Audit a merge table for unreachable tokens

Trailing whitespace and prompt boundaries

Token healing

The multilingual token tax

Code and whitespace

Train-serve tokenizer mismatch

Read more

Watch next