Why a chat model miscounts the r's in strawberry, why Tamil costs twelve times more than English, and how the tokenizers of GPT, BERT, and Llama actually work, built from scratch.

Ask a chat model how many times the letter r appears in "strawberry" and for a long time even the best-known chat models answered two. It sounds like a reasoning failure. It is mostly a reading failure. GPT-4's tokenizer turns the question "How many r's are in strawberry?" into eight integers, and one of them, number 73700, stands for the whole chunk " strawberry", leading space included. The model never sees the letters s, t, r, a, w, b, e, r, r, y. It sees one opaque id, the same way you might see a single Chinese character without knowing how to spell it out in strokes.

Here is a second puzzle. Take the sentence "Where is the nearest train station?" and its Tamil translation. GPT-2's tokenizer spends 7 tokens on the English and 85 on the Tamil, more than twelve times as many. Since API prices, context windows, and compute are all counted in tokens, the Tamil speaker pays twelve times more for the same question and fits a twelfth as much text into the same context window.

Both puzzles have the same root. Before a language model does anything clever, a separate, frozen program called the tokenizer decides what the model is allowed to read. This chapter is about that program. By the end you will be able to build the tokenizers used by GPT, BERT, and Llama from scratch, explain every one of their quirks, and reason about the trade-offs that decide their design.

The model reads integers

Recall from §1.2 that a language model's first layer is an embedding table $\mathbf{E} \in \R^{V \times d}$, with one row per item of a vocabulary $\mathcal{V}$ of size $V$. The model needs a sequence of row numbers. Text is not a sequence of row numbers, so something has to translate. The full pipeline looks like this.

"Hello world"  --encode-->  [9906, 1917]  --lookup-->  E[9906], E[1917]  -->  Transformer
                                                                                  |
"Hello world!" <--decode--  [9906, 1917, 0]  <--sample--  logits over V  <-------+

The tokenizer owns two functions. encode turns a string into a list of token ids $w_1, \dots, w_T$, each an integer in $[0, V)$. decode turns a list of ids back into a string. The model sits between them and only ever sees ids. It predicts a probability distribution over the $V$ possible next ids, and the decoder turns the chosen id back into text.

Three facts about this arrangement shape everything that follows.

1. The tokenizer is trained separately. It has its own training data and its own algorithm (usually byte pair encoding, §2.2). It is trained once, before the model, and then frozen. Andrej Karpathy likes to call it a completely separate stage of the pipeline, and that separation is the source of many bugs, as we'll see in §2.6.
2. The vocabulary fixes the shape of the model. $V$ sets the number of rows of the input embedding table and the number of outputs of the final softmax. Change the tokenizer and you must retrain (or at least resize) the model.
3. The tokenizer decides what is cheap. Everything a model does is paid for per token. A tokenizer that needs fewer tokens for the same text makes the model faster, cheaper, and able to fit more text in its context window.

We also want one guarantee. For any string $s$, decoding its encoding must give $s$ back,

In plain words, the tokenizer must never lose information. A tokenizer that maps every unknown word to the same <unk> id breaks this guarantee, and we'll see that modern tokenizers go to some lengths to keep it. The reverse direction, $\operatorname{encode}(\operatorname{decode}(\mathbf{w})) = \mathbf{w}$ for an arbitrary id sequence $\mathbf{w}$, does not hold in general, and that asymmetry will matter in §2.6.

What the model on the other side does

You don't need to know how a language model works inside to understand tokenizers. But later sections of this chapter talk about generation, masks, padding, and positions, so here is the minimum, in one place. Chapter 3 builds the idea properly, starting from language modeling as prediction, and Chapters 4 to 6 build the network itself.

It predicts the next token. A language model reads the ids $w_1, \dots, w_t$ and outputs a probability for every one of the $V$ ids that could come next, $p(w_{t+1} \mid w_1, \dots, w_t)$. Inside, each id becomes its row of $\mathbf{E}$, a network mixes the rows of the context into one vector $\mathbf{h} \in \R^d$ for the current position, and an output matrix $\mathbf{E}_{\text{out}} \in \R^{V \times d}$ turns $\mathbf{h}$ into one score per vocabulary entry, the logits $s_i = \mathbf{E}_{\text{out},i}^\top \mathbf{h}$. A softmax turns the logits into probabilities. Training lowers the cross-entropy of the token that actually came next (§0.5), averaged over every position of every training text. In every model this chapter names, the mixing network is a Transformer.

It writes one token at a time. To generate, the model predicts a distribution, picks or samples one id from it, appends that id to the input, and repeats. It stops when it produces a special end-of-sequence id, or when it reaches a length limit. How to pick from the distribution is the subject of §6.4.

It sees a window of numbered positions. A model reads at most a fixed number of tokens at once, its context window, from a few thousand to over a million tokens in current models. The mixing step does not by itself know the order of the rows it combines, so each token is also told its position, 0, 1, 2, and so on, usually through a position vector added to its embedding (§5.4 covers the ways to do this).

It is told which positions it may look at. When predicting the token after position $t$, the model may only use positions up to $t$, or predicting the next token would be cheating. And when sequences of different lengths are stacked into one batch, the shorter ones are filled out with filler tokens that must be ignored. Both rules are written as a 0 or 1 per position, the attention mask, which §4.4 builds in detail.

The simplest model with all of these parts is a table of next-token counts. It has no network, only the predict, pick, append loop and a stop token, but every language model in this book runs the same loop.

After "cat" the table gives "sat" probability 2/3 and "ate" 1/3. Two of the three generated lines, "a cat ate" and "the cat sat down", never occur in the corpus, and each line ends when <eos> is sampled. This model only looks one token back. Real models look back over the whole context window, which is what the rest of the book builds.

Three ways to cut text

Before learning any algorithm, let's look at the design space. There are three natural ways to cut a string into units.

Characters. Every character is a token. The vocabulary is small (a few hundred characters cover most European text), and no word is ever unknown. But sequences are long. English averages five to six characters per word once you count the space and punctuation, so a 1,000-word essay becomes about 5,800 tokens.

Words. Split on spaces and punctuation, and make every word a token. Sequences are short, about one token per word. But the vocabulary is enormous, and it is never complete. Every new name, typo, number, hashtag, or inflection ("unembeddable") is a word the vocabulary has never seen.

Subwords. Keep frequent words whole and break rare words into frequent pieces, so "the" is one token while "unembeddable" becomes something like "un", "embed", "dable". This is what every modern language model does. The vocabulary is a fixed size you choose, typically 32,000 to 256,000, and any string can still be written down.

Try all three on your own text below. The subword tokenizer here is a small byte pair encoding trained on a few pages of English (you'll build exactly this in §2.2). Slide the number of merges and watch it move from characters toward words.

A few hundred merges already get you to about two characters per token on new text (and close to three on the pages the tokenizer was trained on), while the word-level tokenizer marks every word it never saw as unknown. Real tokenizers with 50,000 to 200,000 pieces reach about four characters per token on English, as we'll measure below.

Why words fail, Zipf and Heaps

Why not just use a really big word vocabulary? Because natural language has a long tail that never ends. Word frequencies follow Zipf's law. The frequency of the word of rank $k$ (the $k$-th most common) falls roughly like $k^{-a}$, with an exponent $a$ close to 1. A few words are extremely common, and a vast number are extremely rare. Heaps' law is the other side of the same coin. The number of distinct word types you have seen after reading $n$ words keeps growing like

In plain words, every new page of text brings new words, forever, just more slowly. For English news text, Manning, Raghavan and Schütze report $\beta \approx 0.49$ on the Reuters RCV1 collection, and both laws are empirical findings, not theorems.

So any fixed word vocabulary has an out-of-vocabulary (OOV) rate, the fraction of running words it cannot represent. The standard word-level fix maps every such word to one shared <unk> vector, which, as §1.9 showed, throws away everything about the word. The cell below makes this concrete on a synthetic corpus whose word frequencies follow a Zipf profile with exponent $a = 1.3$.

On this corpus a vocabulary of 1,000 words misses 10.6% of held-out words, and even keeping every one of the 15,703 types seen in training still misses 6.1%. No vocabulary size can fix that floor, because those words did not occur in training. Real English grows its vocabulary more slowly than this toy corpus (a Heaps exponent near 0.5 rather than the 0.73 fitted here), so its OOV rates are lower, but the shape is the same, and it is far worse for morphologically rich languages like Finnish or Turkish, where one root can take thousands of inflected forms.

def oov_rate(train_tokens, test_tokens, max_vocab):
    """Fraction of test tokens (running words) not in the top-max_vocab training words."""
    raise NotImplementedError

Why characters fail, the cost of length

Characters have no OOV problem, so why not use them? Because the model pays for every position. Let's count with a sentence.

The character sequence is about six times longer than the word sequence (166 positions versus 27). Three costs grow with the length $T$.

• Compute. Each Transformer layer does work proportional to $T$ for its projections and feed-forward networks, plus work proportional to $T^2$ for attention (§4.7 counts this precisely in What attention costs). Six times the length means about six times the linear work and up to 36 times the attention work.
• Context. A model with a context window of 8,192 positions holds 8,192 characters, a few pages, instead of the roughly 33,000 characters that 8,192 subword tokens would hold.
• Learning. A character-level model must first learn to assemble letters into words before it can learn anything about meaning. It spends capacity and data on spelling.

Subwords are a compromise between these failure modes. They keep sequences short where text is predictable (frequent words are single tokens) and fall back to smaller pieces where it is not (rare words, names, typos), so nothing is ever unknown. Sennrich, Haddow and Birch (2016) introduced the idea for machine translation, where rare words and names had been a chronic problem, and it took over the field.

Measuring a tokenizer

To compare tokenizers we need numbers. Let a text have $n_b$ bytes of UTF-8 (you'll meet UTF-8 properly in §2.3; for plain English it is one byte per character), $n_w$ whitespace-separated words, and let a tokenizer turn it into $T$ tokens. Three standard metrics follow.

In plain words, $\rho$ is the compression ratio, how much raw text each token carries on average, and higher is better. Fertility $\phi$ is how many pieces a typical word shatters into, and its minimum is 1, reached when every word is a single token. The last one, the proportion of continued words, says how often words are split at all. Rust et al. (2021) used fertility and continued-word proportion to show that multilingual models over-segment languages like Arabic, Finnish, Korean, Russian, and Turkish, and that giving a model a dedicated monolingual tokenizer improves its downstream performance in almost every task and language they tested.

Some people report characters per token instead of bytes per token. For English the two are nearly equal. For other scripts they are very different, because a Hindi or Chinese character takes three bytes in UTF-8, so always say which one you mean.

Here are measured bytes per token for real tokenizers, on the full text of two public-domain novels (Alice's Adventures in Wonderland and The Adventures of Tom Sawyer, 556,735 bytes), counted with OpenAI's tiktoken library.

Doubling the vocabulary from GPT-4's to GPT-4o's barely moves English. Most of the new o200k entries serve other languages, as §2.6 shows. Meta reports the same lever from the other direction. Moving Llama 3 to a 128K vocabulary improved its compression on English from 3.17 to 3.94 characters per token compared with the Llama 2 tokenizer (Llama Team 2024), which means 24% more English text fits in the same context window, and a fixed compute budget covers 24% more text.

def tokenizer_stats(segmented):
    """segmented: list of words, each a list of token strings."""
    raise NotImplementedError

Comparing models that use different tokenizers

Here is a trap that catches researchers, not just beginners. Model A reports a validation loss of 2.0 nats per token. Model B, with a different tokenizer, reports 2.3. Which model is better?

You cannot tell from those numbers. A loss per token depends on how much text a token holds. A tokenizer with long tokens makes each prediction harder (more information per token) but needs fewer predictions. The fair comparison is the loss of the whole text, which is the same no matter how you cut it up. Recall from §0.5 that the cross-entropy loss is an average negative log-probability. A language model predicts each token from the ones before it, so by the chain rule of probability (§0.4) it assigns the whole text the log-probability $\log p(\text{text}) = \sum_{t=1}^{T} \log p(w_t \mid w_{<t})$, where $w_{<t}$ is shorthand for $w_1, \dots, w_{t-1}$. Then

In plain words, multiply the per-token loss by the number of tokens to get the total, divide by the number of bytes, and convert nats to bits by dividing by $\ln 2$. Since $T/n_b = 1/\rho$, bits per byte is just the per-token loss divided by the compression ratio (and by $\ln 2$).

Now finish the example. Suppose A's tokenizer gives $\rho_A = 4.0$ bytes per token and B's gives $\rho_B = 4.8$. Then A achieves $2.0 / (4.0 \ln 2) = 0.721$ bits per byte and B achieves $2.3 / (4.8 \ln 2) = 0.691$. Model B is the better model, despite its larger per-token loss. This is why papers that compare tokenizers, or byte-level models against subword models (§2.7), report bits per byte. Per-token perplexity, the exponentiated loss of §0.5, is only comparable between models that share a tokenizer. We'll use this again when we evaluate language models in §6.6.

There is one catch in the formula. The model's first token is predicted from nothing (or from a beginning-of-sequence token), so implementations differ slightly in what they count. For long texts the difference vanishes.

import math

def bits_per_byte(loss_per_token, n_tokens, n_bytes):
    raise NotImplementedError

def better_model(a, b):
    raise NotImplementedError

What a token costs

So far tokens have cost compute. They also cost money and memory, in ways worth having at your fingertips.

• Inference. A Transformer with $N$ parameters spends about $2N$ floating-point operations per token in a forward pass, because each weight takes part in one multiplication and one addition for every token it processes (§5.6 does the full count, attention included). The cost of processing a document is therefore about $2N \cdot n_b / \rho$. A tokenizer that packs 20% more bytes into each token needs $1/1.2 \approx 0.83$ times as many tokens, so processing the same text costs about 17% less, before any other optimization.
• Training. Training cost is about $6N$ FLOPs per token, and scaling laws are written in tokens. A "2 trillion token" dataset under one tokenizer is a different amount of text under another.
• Parameters. The embedding table has $V \times d$ parameters and, unless its weights are tied with the output layer (the weight tying of §1.2), the output softmax has another $V \times d$. For small models this is a large fraction of all parameters, which §2.7 quantifies.
• Money and context. API providers charge per input and output token and size context windows in tokens. The same request costs very different amounts in different languages (§2.6), and the same document fits or doesn't fit depending on the tokenizer.

The real way

In practice you never write the tokenizer of a production model yourself. You load the exact one the model was trained with. OpenAI's tiktoken and Hugging Face's tokenizers library are the two you'll meet most.

And here is the other end of the pipeline in PyTorch. The simplest possible tokenizer, raw UTF-8 bytes with $V = 256$, feeds an embedding table, a stand-in for the Transformer, and a tied output layer. The loss comes out per token, and we convert it to bits per byte exactly as derived above.

A model that only sees the previous byte cannot do much better than a bigram model, which is the point. Everything interesting comes from the layers we'll build in Chapters 4 to 6. What matters here is the plumbing. Strings become ids, ids become rows, and the loss you report depends on what a token is.

Read more

• Sennrich, Haddow & Birch (2016). Neural Machine Translation of Rare Words with Subword Units. The paper that brought subword units to neural models. Its introduction is still the clearest statement of the open-vocabulary problem.
• Karpathy (2024). minbpe. Minimal, readable code for the tokenizers of the GPT series, and the companion to his video for this chapter.
• Rust et al. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Fertility and continued-word proportion, and evidence that the tokenizer matters as much as the pretraining data.
• Llama Team, Meta (2024). The Llama 3 Herd of Models. Section 3.2 explains the 128K-token vocabulary and the compression gain over Llama 2.
• Manning, Raghavan & Schütze (2008). Introduction to Information Retrieval, §5.1. Heaps' law and Zipf's law, with fits on a real collection.
• Stanford CS336 (2025). Language Modeling from Scratch, assignment 1. Build and benchmark a byte-level BPE tokenizer as the first step of a full language model.

Watch next

▶

▶

▶