Llama 2 had a vocabulary of 32,000 tokens. Llama 3, a year later, had 128,256. GPT-4o went to 200,019, and Gemma uses 256,000. At the other extreme, the mT5 models use a 250,000-token vocabulary, and in the smallest of them, mT5-Small, 85% of all parameters sit in the input embedding and output softmax, leaving a sliver for the Transformer that does the actual thinking (Xue et al. 2022 report this in their comparison table). So what is the right vocabulary size? And a more radical question. If tokenizers cause all the trouble of §2.6, why not delete them and let the model read bytes?

This section answers both. First we price a vocabulary, in parameters, compute, memory, and compression, and find the optimum in a simple model. Then we look at the models that read bytes directly, ByT5, MegaByte, and the Byte Latent Transformer, and end with a preview of how images and audio become tokens.

What a vocabulary costs

Let the model have width $d$, $L$ layers, and a vocabulary of $V$ tokens. Its token-related parts are the input embedding table $\mathbf{E} \in \R^{V \times d}$ and the output matrix $\mathbf{E}_{\text{out}} \in \R^{V \times d}$ that produces the logits (the same matrix, if the weights are tied). The Transformer between them, the body, has roughly $12 L d^2$ parameters. Take that count on trust for now (§5.6 derives it block by block). Nothing below depends on what is inside the body, only on its size. Four costs grow with $V$.

Parameters. The vocabulary holds $Vd$ parameters if tied, $2Vd$ if untied. For small models this dominates.

Compute. The input embedding is a lookup and costs nothing, but the output layer multiplies the $d$-dimensional hidden state by the $V \times d$ matrix $\mathbf{E}_{\text{out}}$ at every position. That is one multiplication and one addition per entry, $2Vd$ floating-point operations per token, next to about two per body parameter, $2 \cdot 12Ld^2$, for the body (the rule of thumb from §2.1, which leaves out attention's extra cost on long contexts). For Llama 3 8B the output layer is about 7% of the forward pass. At inference, sampling also needs a softmax over $V$ entries per step.

Memory. Training materializes a logit for every token in the batch and every vocabulary entry. At 8,192 tokens per batch and $V = 128{,}256$, the logits alone take $8192 \times 128256 \times 4 \approx 4.2$ GB in 32-bit floats, and their gradient as much again. This is why implementations that compute the cross-entropy without ever storing the full logit matrix are popular for large vocabularies. The simplest version processes the positions in chunks, as in the block below, and production versions fuse the matrix multiply and the loss into one GPU kernel (§11.7 explains kernel fusion).

The losses and gradients agree to float32 rounding, while at most one block of logits exists at a time. The checkpoint call is what makes that true in the backward pass too. Without it, autograd would keep every block's logits for the backward pass, and the saving would be lost. The price is computing each block's logits twice.

Data per token. Token frequencies follow Zipf's law, so a bigger vocabulary adds ever rarer tokens. A token that appears once in ten million positions gets very few gradient updates, and its embedding may stay poor or become one of the under-trained tokens of §2.6.

Against all of this stands one benefit. A larger vocabulary compresses better, so the same text becomes fewer tokens, and every other cost of the model is paid per token. To see the size of this effect, we trained byte-level BPE tokenizers of every size from 512 to 32,768 tokens on six public-domain novels and measured bytes per token $\rho$ on two others.

Each doubling of the vocabulary helps less than the one before. From 2K to 16K, each doubling gains about 0.83 times as much as the previous one (the last measured doubling shrank a little faster), and extrapolating at 0.83 gives about 4.1 bytes per token at 256K, close to what GPT-4's and GPT-4o's tokenizers achieve on the same two novels (4.11 and 4.13), though they were trained on far more text, so treat the agreement as a sanity check rather than a law.

Choosing the vocabulary size

Put the pieces together. Processing $n_b$ bytes of text takes $n_b / \rho(V)$ tokens, and each costs about $2(N + Vd)$ FLOPs in the forward pass, where $N \approx 12Ld^2$ counts the body. The compute per byte is

In plain words, the numerator grows linearly with $V$ through the output layer, and the denominator grows slowly through compression. The best vocabulary balances the two. Take the logarithm, $\log C = \log 2 + \log(N + Vd) - \log \rho(V)$, and set its derivative with respect to $V$ to zero, writing $\rho'(V)$ for the derivative of $\rho$,

In plain words, grow the vocabulary until the relative gain in compression from one more token equals the relative increase in compute per token. The left side falls as $V$ grows (diminishing returns), and a bigger body $N$ makes the right side smaller, so the crossing point moves to larger $V$. Bigger models want bigger vocabularies.

The optimum moves from about 6,000 tokens for a 25-million-parameter model to about 33,000 for a 1.2-billion-parameter model and about 220,000 for a 64-billion-parameter model. Notice also how flat the curves are near their minima. Being off by a factor of two in $V$ costs only a few percent of compute, which is why real vocabularies vary so much without disaster.

This toy model counts only compute per byte. It ignores that larger vocabularies spend more parameters on rarely updated rows, that the softmax has a memory cost, and that the model's quality per FLOP also depends on $V$. Tao et al. (2024) did the careful version, fitting scaling laws over models from 33M to 3B parameters with different vocabularies, and reached the same qualitative conclusion, that the compute-optimal vocabulary grows with the compute budget and that most models use vocabularies that are too small. They predict that Llama 2 70B should have used at least 216K tokens instead of 32K, and they show that, at a fixed budget of $2.3 \times 10^{21}$ FLOPs for a 3B model, raising the vocabulary from 32K to 43K improved ARC-Challenge accuracy from 29.1 to 32.0. Our toy lands in the same range for a model of that size, which is encouraging but partly luck.

Two practical considerations also set $V$. Multilingual coverage needs many more tokens than English alone (§2.6), and vocabulary sizes are usually rounded to a multiple of 64 or 128 so that the embedding and softmax matrices map efficiently onto GPU hardware.

def vocab_costs(V, d, L, tied, rho):
    raise NotImplementedError

def best_vocab(Vs, rhos, d, L):
    raise NotImplementedError

Reading bytes directly

The other answer to "how big should the vocabulary be?" is "256". A byte-level model has no tokenizer to train, no unknown inputs, no glitch tokens, no multilingual merge budget, and it sees every character, so spelling and character tasks are no longer hidden. The price is length. For English, sequences become about four times longer than with a good subword tokenizer, which multiplies the per-position compute by four and the attention compute by up to sixteen.

ByT5 (Xue et al. 2022) showed that a standard Transformer can work on bytes with modest changes. Starting from mT5, they replaced its 250,000-token vocabulary with 256 byte ids plus a few special tokens and moved the parameters this freed into the Transformer body, so vocabulary parameters drop from 85% of mT5-Small to 0.3% of ByT5-Small. (They also gave the encoder more layers than the decoder of this encoder-decoder model and made the pretraining task hide longer spans of text, adjustments that make sense once those parts have been built.) ByT5 was competitive with mT5, clearly more robust to noise such as typos and random casing, and better on tasks sensitive to spelling and pronunciation. It was also slower, because mT5's tokens hold about 4.1 bytes on average in their pretraining data. ByT5 ran about as fast as mT5 on a short-input pronunciation task, 1.5 to 2.6 times slower on transliteration, 3.7 to 6.4 times slower on document summarization, and 6.4 to 9.5 times slower on XNLI classification. Earlier encoders such as CANINE (Clark et al. 2022) and Charformer (Tay et al. 2022) read characters but downsampled them early with convolutions or learned pooling, trading resolution for speed.

The lesson from ByT5 is that bytes work, and that the remaining problem is cost. The next two models attack the cost by grouping bytes into patches that the expensive part of the model processes as units.

MegaByte, fixed-size patches

Yu et al. (2023) split a sequence of $T$ bytes into $K = T/P$ patches of $P$ bytes each, and use two Transformers. A large global model reads one vector per patch (built from the patch's $P$ byte embeddings) and carries the context from patch to patch. A small local model then predicts the bytes inside each patch one at a time, from the bytes of that patch so far and the global model's summary of all earlier patches. The expensive model therefore sees $T/P$ positions instead of $T$. What the two networks compute inside is a preview, since it needs the Transformer of Chapters 4 and 5. Here we only need one fact about them, which §4.7 derives, that attention over $n$ positions costs work proportional to $n^2$, because every position compares itself with every other.

Count that cost. The global model attends over $K = T/P$ positions, costing $T^2/P^2$. The local model runs on $T/P$ patches of length $P$, costing $P^2$ each, so $TP$ in total. Minimizing the sum over $P$,

In plain words, small patches make the global sequence long and large patches make the local sequences long, and the balance point at a patch size near the cube root of the length turns quadratic attention into the four-thirds power. For a million-byte sequence that is about 126 bytes per patch and roughly 5,000 times less attention work than full attention over every byte. The attention count is not the whole story, since most of a large Transformer's compute is in the parts that process each position separately, and MegaByte saves there too by running its big model once per patch rather than once per byte.

import numpy as np

def megabyte_inputs(x, P, pad):
    raise NotImplementedError

The Byte Latent Transformer, patches that follow the data

Fixed patches waste compute. A patch of eight bytes inside "the the the" deserves far less thought than a patch of eight bytes that starts a new name. Pagnoni et al. (2024) made patch boundaries depend on how hard the next byte is to predict. A small byte-level language model (100 million parameters by default) computes, at every position $t$, the entropy of its prediction for the next byte,

In plain words, $H(x_t)$ is high when the small model is unsure what comes next, typically at the start of a word or a new idea, and low inside predictable stretches. BLT starts a new patch at byte $t$ when the entropy crosses a global threshold, $H(x_t) > \tau_g$, or, in a second variant, when it jumps relative to the previous byte, $H(x_t) - H(x_{t-1}) > \tau_r$, which detects breaks in an otherwise decreasing run of entropy. Predictable text becomes long patches and surprising text short ones, so the big model spends its compute where the information is.

Around the patches, BLT has the same shape as MegaByte. Small byte-level networks turn each patch's bytes into one vector and, at the other end, turn the big model's patch outputs back into next-byte predictions, while a large latent Transformer runs over the patch vectors. Those small networks use cross-attention, which §4.6 introduces, so we leave their internals as a preview. The patching rule itself needs nothing beyond entropy. Here it is with a tiny byte n-gram model standing in for BLT's small language model.

Phrases the small model has seen before ("morning the", "wrote down the") collapse into long patches, while the new words ("names", "strange") are cut into short ones. You can try your own text and thresholds below.

How well does it work? In the first FLOP-controlled scaling study of byte-level models, up to 8 billion parameters and 4 trillion training bytes, BLT matched a tokenizer-based Llama 3 architecture trained on the same data (an average of 61.1 versus 60.0 over seven benchmarks in the paper's FLOP-matched comparison), and the authors report that it can use up to 50% fewer inference FLOPs by choosing larger patches, with some loss in quality at the largest patch sizes. Its byte-level view also shows. On a benchmark of character-level manipulation (CUTE), the 8B BLT scored 54.1 against 27.5 for the tokenizer-based model trained on the same data, and on its spelling subtask 99.9 against 1.1. The patching model is a separate, small network, and the paper notes that its implementation is not yet as fast in wall-clock time as heavily optimized tokenizer-based models.

import numpy as np

def entropy_patches(H, tau_g=None, tau_r=None):
    raise NotImplementedError

def patch_lengths(starts, n):
    raise NotImplementedError

Where tokenizer-free models stand

Other designs explore the same space. SpaceByte (Slagle 2024) places patch boundaries at spaces and other word boundaries, a cheap rule that works well for languages that use spaces. MambaByte (Wang et al. 2024) runs a state-space model, whose cost grows linearly in length, directly over bytes, the family of models that replaces attention with selective state spaces. The common thread is that the model itself, not a frozen preprocessing step, decides how to group bytes.

As far as public descriptions go, most widely used frontier models still run on subword tokenizers, typically byte-level BPE with 100K to 256K tokens. Tokenizers are fast, well understood, and good enough once the vocabulary is large, and the systems that serve models, from KV caches (§6.5) to speculative decoding, assume a token is a fixed unit. Byte-level models with learned patching are the most credible path to deleting the tokenizer, and BLT shows the scaling is there. Whether they take over is an open question, and a good one to have an opinion about in an interview.

Tokens for images and audio

Everything in this chapter has been about text, but the same question, what is the unit the model reads, comes back for every other modality.

• Images as patches. A Vision Transformer cuts an image into a grid of patches and linearly projects each one into a vector, as you saw in §1.8. These are continuous tokens, with no vocabulary at all, and the way they scale with resolution is the central design question for Vision Transformers.
• Images and audio as codebook ids. A vector-quantized autoencoder learns a codebook of a few thousand vectors and replaces each patch of an image, or each frame of audio, with the id of its nearest code. Those ids are discrete tokens that a language model can read and write exactly like words, which is how image tokenizers let a Transformer generate pictures.
• Audio with stacked codebooks. Neural audio codecs quantize each frame with several codebooks in sequence, each coding the residual error of the previous one, which gives high-quality sound at a few kilobits per second. Language models over these codec tokens are the basis of modern voice models.
• Mixing modalities. Vision-language models turn image patches into vectors in the language model's embedding space, so a picture becomes a few hundred "tokens" between ordinary text tokens, the design of vision-language connectors.

The trade-offs you now know carry over directly. A tokenizer for images faces the same tension between vocabulary size and sequence length, the same question of how much detail a token should hold, and the same choice between fixed patches and patches that adapt to the content. Karpathy's tokenizer lecture closes with this point, and §1.10 drew the same map from the embedding side.

Read more

• Tao et al. (2024). Scaling Laws with Vocabulary, Larger Models Deserve Larger Vocabularies. Three ways to predict the compute-optimal vocabulary size, and the 216K estimate for Llama 2 70B.
• Xue et al. (2022). ByT5, Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. The byte-level T5, with a careful accounting of parameters, compute, and speed.
• Yu et al. (2023). MegaByte, Predicting Million-byte Sequences with Multiscale Transformers. Global and local models over fixed patches, and the four-thirds power cost.
• Pagnoni et al. (2024). Byte Latent Transformer, Patches Scale Better Than Tokens. Entropy-based patching and the first FLOP-controlled scaling study of byte-level models.
• Slagle (2024). SpaceByte, Towards Deleting Tokenization from Large Language Modeling. Patches at word boundaries, a simpler alternative to learned patching.
• Clark et al. (2022). CANINE, Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Character-level encoding with downsampling.

Watch next

▶

▶