Everything from the earlier chapters, assembled into one working language model. You will prepare the data, write the model, train it, sample from it, make it fast, measure it, and see exactly what changes on the way to a real one.

This chapter builds a GPT, the same kind of model as GPT-2, and every line of it is ours. Over eight sections you'll write the data pipeline, the model, the training loop, the sampler, the KV cache, and the evaluation code, see exactly what changes when the model is a thousand times bigger, and find out what the finished model can do from its prompt alone. Everything earlier in the book feeds into it, from tokenizers and attention to the Transformer block.

We start with the least glamorous piece, and the one where the silent bugs live. A GPT never sees a poem, a sentence, or a file. It sees long rows of integers, and everything it learns about where documents begin and end, and about what counts as training and what counts as test, comes from decisions we make before the model exists.

Here's a puzzle to keep in mind. You train a small model and its loss falls to almost exactly zero within a hundred steps, on the training set and on the validation set. Is that a great model? Almost certainly not. Two different one-line bugs in the data pipeline produce exactly that curve. By the end of this section you'll be able to name both, and a third bug that makes validation loss look better than it is.

Our corpus, eight sonnets

We'll train on a corpus small enough to read in two minutes and big enough to learn from, eight of Shakespeare's sonnets (numbers 1, 18, 29, 30, 55, 60, 73, and 116, from the public-domain Project Gutenberg edition). That's about 4,900 characters. Real pretraining corpora are millions to billions of times larger (§8.2 shows where they come from), but every step of the pipeline is the same.

From characters to integers

The model needs integers, so the first decision is the tokenizer. §2.3 built the real one, byte-level BPE, which GPT-2 uses with a vocabulary of $V = 50{,}257$ (256 byte tokens, 50,000 merges, and one special token). For this chapter we use the simplest tokenizer there is. Every distinct character is a token.

Why characters? Three reasons. Every token is readable, so you can check the pipeline by eye. The vocabulary is tiny, so the embedding table and the output softmax cost almost nothing. And the model trains in seconds on a CPU. The price is length. A character model needs about four times as many positions as GPT-2's tokenizer for the same English text, so its context window covers about a quarter as much text, and attention, whose cost grows with the square of the length (§4.7), pays for it. Nothing else in this chapter depends on the choice. Swap in the BPE tokenizer from §2.2 and every function below works unchanged.

We add one special token, <|endoftext|> (we'll write EOT), the same separator GPT-2 uses. It's not a character. It's an id with no spelling, and we'll put one before every document. Its job is to tell the model "what came before is unrelated".

Notice that stoi has no entry for EOT. If a document happens to contain the literal string <|endoftext|>, it is encoded as thirteen ordinary characters, never as the special id. Only our pipeline can insert the real separator. Production tokenizers make the same distinction on purpose (tiktoken refuses to encode special-token strings unless you allow it), because otherwise anyone who can put text in front of the model can forge a document boundary. §2.5 covers special tokens in depth.

def build_tokenizer(docs):
    """returns encode, decode, V, eot"""
    raise NotImplementedError

Split by document, never by position

Before any training we set aside text the model will never train on, so we can measure how well it does on text it hasn't seen. The question is how to cut.

The tempting way is to shuffle positions, or to take every tenth character, or every tenth window. That is a leak. Neighboring characters are almost perfectly predictable from each other, so a validation character whose neighbors are in the training set has effectively been seen. Validation loss comes out nearly equal to training loss, and you conclude the model generalizes when it has only memorized. This is the third bug from the puzzle.

The rule is to split at the level of the unit you want to generalize across. For a language model that unit is the document. We hold out the eighth sonnet (116) and train on the other seven. At scale the same rule becomes "split by document, then remove near-duplicates across the split", because the web contains many copies of the same text (§8.3 covers deduplication and benchmark contamination).

Each split becomes one long stream of ids, with an EOT before every document and one at the very end.

Putting EOT before each document (not only after) matters for generation. The model then learns what tends to follow EOT, which is the opening of a sonnet. To generate a fresh sonnet later, we'll prompt it with a single EOT. OpenAI's GPT-2 code starts its unconditional samples the same way.

The one-line trick, shifting by one

A language model assigns a probability to a whole sequence by the chain rule of probability (§3.1),

where $w_t$ is the token at position $t$ and $n$ is the number of tokens. In plain words, the probability of a text is the probability of its first token, times the probability of the second given the first, and so on. So training a language model means training one thing, a predictor of the next token from all the tokens before it.

Take any window of $T + 1$ consecutive tokens $s_0, \dots, s_T$ from the stream. Split it into an input and a target that is the same text moved one step left,

The training loss for the window averages the negative log-probability of every target,

In plain words, position $t$ reads the tokens up to and including $x_t$ and must predict $y_t$, the token right after it. The model with parameters $\theta$ produces all $T$ predictions in one forward pass.

That last sentence is the trick that makes GPT training efficient. Because the causal mask (§4.4) stops position $t$ from looking at anything after $x_t$, one window of length $T$ is really $T$ training examples, with contexts of length 1, 2, up to $T$, computed together for roughly the price of one. Here's what one window of seven characters from sonnet 18 contains.

Now the puzzle has two of its answers. If you forget the shift and train with y = x, position $t$ is asked to predict $x_t$, which it can see. Loss goes to zero in a few steps, and the model has learned to copy. If you shift the wrong way, y[t] = x[t-1], the target is again visible, with the same result. Both bugs give a perfect-looking curve on training and validation data, which is how you recognize them. A real language model on real text never gets close to zero loss, because text is genuinely uncertain.

Context windows and random crops

The model has a maximum context, the block size $T$. It exists for two reasons. GPT-2 has a learned position embedding for each position $0, \dots, T-1$ (§5.4), so there is no vector for position $T$. And attention costs grow as $T^2$. GPT-1 used $T = 512$, GPT-2 used 1024, and GPT-3 used 2048.

So we train on windows of length $T$ cut from the stream. The simplest sampler picks a uniformly random start $i$ for each row of the batch. The last target of a window starting at $i$ is token $i + T$, which must exist, so with $n$ tokens in the stream

That range is where off-by-one bugs live. With np.random.default_rng().integers(0, n - T), the upper bound is exclusive, so you get exactly the right set. Write n - T + 1 and once in a while the target row comes back one token short, which crashes, or worse, gets silently padded. Write n - T - 1 and the very last window can never be chosen, which nobody notices.

How often is each token used?

Random crops don't use every token equally. A target at stream position $j$ (for $1 \le j \le n-1$) is inside the window starting at $i$ exactly when $i + 1 \le j \le i + T$, so the number of windows that contain it is

In plain words, every token is used equally often, except the first $T$ and the last $T$ of the stream, which ramp down linearly to almost never. With one stream of millions of tokens this edge effect is negligible. It matters when you crop inside short documents instead of across one long stream, because then every document has edges, and the openings and endings of documents, often the most informative parts, get undertrained. Let's check the formula against a simulation.

Random crops have a quiet advantage too. The same token lands at a different position in every window that contains it, with a different amount of context before it. The model must learn to predict from short contexts and long ones, at every position, which is exactly what generation will ask of it.

The alternative is to cut the stream into consecutive, non-overlapping chunks and walk through them in order, so that every token is used exactly once per pass over the data (an epoch). That's what large runs do. They see each token about once, so the randomness comes from shuffling documents and data shards before training, and the loader just reads the next $B \times T + 1$ tokens and reshapes them. Karpathy's GPT-2 reproduction uses exactly this "read the next chunk" loader.

import numpy as np

def sample_batch(data, B, T, rng):
    """returns x, y of shape (B, T), dtype int64, with y shifted one step ahead of x"""
    raise NotImplementedError

Batching, and the numbers at scale

A batch is $B$ windows stacked into two integer arrays of shape $(B, T)$. Each batch carries $B \times T$ predictions. With the tiny settings we'll train with in §6.3 (16 windows of $T = 64$ per step, as two micro-batches of 8) that's 1,024 predictions per step. Here's our sampler, written once for the rest of this section.

Real runs are big enough that the storage format matters. Ids are stored as uint16, because GPT-2's 50,257 ids fit under 65,536. That's 2 bytes per token instead of 8 for int64, so ten billion tokens fit in 20 GB, read straight from disk with a memory map and converted to int64 one batch at a time. (nanoGPT's data preparation writes exactly such a file.) The batch is described by how many tokens go into each optimizer step,

where $G$ is the number of micro-batches whose gradients are accumulated before each update (§6.3) and $n_{\text{gpu}}$ is the number of data-parallel GPUs. GPT-2 trained with 512 sequences of 1,024 tokens per step. The common reproduction targets $2^{19} = 524{,}288$ tokens per step, for example $B = 64$, $T = 1024$, and 8 GPUs with $G = 1$, or one GPU with $G = 8$.

Document boundaries, packing and masking

Our documents are about 600 characters and our window is 64, but at scale the opposite is common. With $T = 2048$, most web pages are shorter than one window. There are three ways to fill a window, and every production pipeline picks one of them.

Padding. Put one document per row and fill the rest with a pad token that the loss ignores. It's clean, because no document sees another. But it wastes compute on every pad position. A document of length $\ell$ chopped into rows of length $T$ uses $\lceil \ell / T \rceil$ rows and wastes $T\lceil \ell / T \rceil - \ell$ positions.

Packing. Concatenate documents with EOT between them, as we did, and cut windows from the stream. Nothing is wasted, but a window may contain the end of one document and the start of the next, and the second can attend to the first. GPT-3 trained this way. Its paper says documents packed into one sequence were not masked in any special way, only separated by the end-of-text token, which gives the model the information that the context before it is unrelated (Brown et al., 2020). The model learns to mostly ignore what comes before an EOT.

Packing with a document mask. Pack as before, but make attention block-diagonal, so that each token attends only to earlier tokens of its own document, and restart the position ids at each document. You get no waste and no cross-talk. (With learned absolute positions, as in GPT-2, the restart matters, because otherwise a document that begins mid-window is read at the wrong positions. With rotary embeddings (§5.5) attention scores depend only on the offset between two tokens, so once the mask keeps documents apart, restarting the positions changes nothing.) The cost is an attention kernel that accepts variable-length segments (FlashAttention's variable-length interface, or PyTorch's FlexAttention, both in §12.4). LLaMA 3 trained with such a mask and reported that it had limited effect in standard pretraining but mattered for continued pretraining on very long sequences (Dubey et al., 2024).

With our 600-character sonnets and $T = 2048$, padding wastes about 70 percent of every batch. Real corpora have a long tail of short documents, which is why nearly everyone packs.

import numpy as np

def packed_mask(x, eot):
    """returns mask (T, T) bool and pos (T,) int"""
    raise NotImplementedError

In PyTorch

The PyTorch version stores ids compactly, gathers each batch with one indexing operation, and adds the two lines that matter on a GPU. Pinned (page-locked) host memory lets the copy to the GPU run asynchronously, overlapping with compute.

buf[1:].view(B, T) is worth a second look. Because the loader reads $B \times T + 1$ tokens and the next batch starts $B \times T$ later, the last target of each batch is the first input of the next one. No token is lost between batches, and none is a target twice.

Read more

• Radford et al. (2019). Language Models are Unsupervised Multitask Learners. GPT-2, its WebText data, byte-level BPE, and the 1,024-token context.
• Brown et al. (2020). Language Models are Few-Shot Learners. GPT-3, including how documents are packed into 2,048-token sequences with end-of-text separators.
• Karpathy. nanoGPT. The prepare.py scripts write uint16 token files and train.py samples random crops from a memory map, just like this section.
• Karpathy. build-nanogpt. The code from "Let's reproduce GPT-2", with the sequential sharded data loader.
• Krell et al. (2021). Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance. Packing algorithms and masks that keep packed documents apart.
• Ding et al. (2024). Fewer Truncations Improve Language Modeling. Why cutting documents at window edges hurts, and best-fit packing to avoid it.
• Dubey et al. (2024). The Llama 3 Herd of Models. A modern pretraining recipe that uses a document mask inside packed sequences.

Watch next

▶

▶