Our GPT has 108 thousand parameters and trains in two seconds on a laptop. GPT-2 small has 124 million, LLaMA-3 8B has 8 billion, and the frontier is far beyond that. What actually changes on the way up? Less than you might think. The training loop of §6.3 is, almost line for line, the loop that trains GPT-2, and the decoding and caching of §6.4 and §6.5 are what serving systems do. Four things change. The recipe's numbers, a handful of architectural changes that make big models more stable and cheaper to run, the systems engineering that makes each step fast, and the arithmetic of the budget, which decides what you can afford at all.

Here's a puzzle that ties them together. By the estimate in Karpathy's nanochat repository, training the full 1.5B GPT-2 cost OpenAI about $43,000 in 2019 (its leaderboard lists the original run at 168 hours). In 2024 his llm.c reproduced the 124M model in 90 minutes for about $20, and by March 2026 nanochat trained a model that matches the full GPT-2's benchmark score in about 1.65 hours on a single node of 8 H100s, roughly $40 of rented time. Where did a factor of about a thousand come from? By the end of this section you'll be able to account for it, since hardware, efficiency, data, and architecture each contribute.

The GPT-2 124M recipe

The best-documented modern recipe for a GPT-2 is the one in Karpathy's "Let's reproduce GPT-2" and its C/CUDA twin llm.c. It is our §6.3 loop with these settings (Karpathy, 2024).

The reproduction slightly beats OpenAI's checkpoint on HellaSwag, 29.9 against 29.4 percent, from just 10 billion tokens. (OpenAI never said how many tokens GPT-2 saw, and for scale, every GPT-3 model trained on 300 billion.) Much of that is credited to better data. FineWeb is filtered far more carefully than 2019's WebText was, and data quality, a theme of §8.2, is worth more per token than almost anything else.

Making one step fast is a separate story, told live in the video. In Karpathy's timings on one A100 (approximate, read off the training log as he goes), a step on 16 sequences of 1,024 tokens went from about 1,000 ms in float32 to 333 ms with TF32 matmuls, 300 ms with bfloat16 autocast, 130 ms with torch.compile fusing kernels (§11.7), 96 ms with FlashAttention (§12.3), 93 ms after padding the vocabulary to 50,304, and 90 ms with a fused AdamW. That's an 11-fold speedup with no change to the math.

Budget arithmetic

Before you train anything big, you compute what it will cost. The workhorse is the rule §5.6 derived. Every weight does one multiply and one add per token going forward, 2 FLOPs, and twice that going backward, because each linear layer computes both the gradient for its input and the gradient for its own weights, each a matmul as large as the forward one. So training on $D$ tokens costs

where $N$ is the number of parameters that each token's computation multiplies through. Two refinements matter. Count the LM head, since $\mathbf{h}\mathbf{E}^\top$ is a real $d \times V$ matmul, but not the embedding lookup, which is free, so for GPT-2 small $N \approx 85.1\text{M} + 38.6\text{M} = 123.7\text{M}$. And attention adds work that no parameter accounts for, the scores $\mathbf{q}\mathbf{K}^\top$ and the weighted sum of values, $4dT$ FLOPs per token per layer going forward and three times that for training, so $12LdT$ per token (counting all $T$ keys, as PaLM's convention does, Chowdhery et al., 2022). For GPT-2 small that's another 15 percent.

In plain words, six FLOPs per parameter per token, plus a correction for attention that grows with the context length.

Model FLOPs utilization

Time follows from the hardware. No GPU runs at its rated speed, so we need a number for the fraction it actually achieves. Model FLOPs utilization (MFU), introduced by the PaLM team (Chowdhery et al., 2022), is the arithmetic the model requires per second divided by what the hardware could do at peak. With $n_{\text{gpu}}$ GPUs of peak throughput $F$ FLOPs per second each,

In plain words, count the FLOPs the math needs, divide by how long the run took, and compare with the hardware's ceiling. Three details make it a precise number rather than a vague one.

• Model FLOPs, not hardware FLOPs. The numerator counts only what the model mathematically requires. Recomputing activations during the backward pass to save memory (activation checkpointing, §10.1) makes the GPU do an extra forward pass, about $2N$ more FLOPs per token, but MFU gives it no credit. Counting that work too gives the hardware FLOPs utilization (HFU), which is always at least as large and can make a slower configuration look busier. MFU was defined to stop exactly that, and §11.2 takes the hardware view.
• The dense peak, in the precision you train in. $F$ is the dense tensor-core peak for your matmul precision, 312 TFLOPS in bfloat16 for an A100 and about 989 for an H100 SXM. Spec sheets also print "with sparsity" figures twice as large, which apply only to matrices with 2:4 structured sparsity, and float32 peaks far smaller. The wrong peak moves MFU by a factor of two or more.
• Whole steps, wall clock. Tokens per second is tokens per step divided by the full step time, including data loading, communication between GPUs, the optimizer, and logging. Anything that stalls the GPU lowers MFU, which is the point.

Work one by hand. Karpathy's llm.c post on the 124M reproduction shows a log line of about 178,000 tokens per second on one A100 (312 TFLOPS peak). Each token needs $6 \times 124\text{M} + 12 \times 12 \times 768 \times 1024 = 7.44 \times 10^8 + 1.13 \times 10^8 = 8.57 \times 10^8$ FLOPs, so the GPU does $8.57 \times 10^8 \times 178{,}000 = 1.53 \times 10^{14}$ useful FLOPs per second, an MFU of $1.53/3.12 = 49$ percent, which is the figure llm.c prints. Even a perfect kernel can't reach 100 percent, because LayerNorm, softmax, GELU, and the optimizer move many bytes per FLOP and run at memory speed, and kernel launches and communication leave gaps. Well-tuned dense training lands at 40 to 60 percent, and a drop of ten points after a code change is a bug worth profiling.

Now check the GPT-2 run. $C \approx (6 \times 123.7\text{M} + 12 \times 12 \times 768 \times 1024) \times 10^{10} \approx 8.6 \times 10^{18}$ FLOPs. Eight A100s at 312 bfloat16 TFLOPS each give $2.5 \times 10^{15}$ FLOPs per second at peak, and finishing in 90 minutes means achieving $1.6 \times 10^{15}$, an MFU of about 63 percent. llm.c reports "up to ~60%", consistent with that.

The LLaMA-2 paper reports 184,320 A100 GPU-hours for the 7B model (Touvron et al., 2023), which these formulas match at an MFU of about 45 percent. The run used 300 tokens per parameter, fifteen times the compute-optimal ratio of about 20 that Chinchilla found (Hoffmann et al., 2022, and §9.3). That's deliberate. A smaller model trained longer costs more to train but less to serve, and a model that will answer billions of queries is cheapest overall when it is small (§9.4).

Memory is the other half of the budget. Mixed-precision AdamW keeps 16 bytes per parameter (2 for the bf16 weights, 2 for the bf16 gradients, 4 for a float32 master copy, and 8 for the two float32 moments), so GPT-2 small needs 2 GB of state and fits on any GPU, while a 7B model needs 112 GB before a single activation, more than any one GPU holds. Sharding that state across GPUs (§10.3) is the first systems problem of real scale.

def training_flops(n_params, n_tokens, n_layers, d_model, seq_len):
    raise NotImplementedError

def mfu(tokens_per_sec, n_params, n_layers, d_model, seq_len, n_gpus, peak_flops):
    raise NotImplementedError

From GPT-2 to a LLaMA-style model

Between GPT-2 (2019) and LLaMA (Touvron et al., 2023), a handful of changes became standard. None changes what the model is, a stack of pre-norm residual blocks with causal attention and an MLP, trained on next-token prediction. Each one fixes a specific weakness, and you should know the reason for each.

RMSNorm instead of LayerNorm. RMSNorm drops the mean subtraction and the bias and divides by the root mean square, $\operatorname{RMSNorm}(\mathbf{x}) = \mathbf{g} \odot \mathbf{x} / \sqrt{\tfrac{1}{d}\sum_j x_j^2 + \epsilon}$ (Zhang & Sennrich, 2019). It works as well as LayerNorm in practice and is cheaper, one reduction instead of two (§5.2).

SwiGLU instead of a GELU MLP. The MLP becomes gated, $\operatorname{MLP}(\mathbf{x}) = \big(\operatorname{SiLU}(\mathbf{x}\mathbf{W}_1) \odot \mathbf{x}\mathbf{W}_3\big)\mathbf{W}_2$, with $\operatorname{SiLU}(z) = z\,\sigma(z)$. Shazeer (2020) found gated units consistently better per parameter. With three matrices instead of two, the hidden width shrinks to about $\tfrac{2}{3} \cdot 4d$ to keep the parameter count near $8d^2$. LLaMA rounds it up to a multiple of 256, which turns $d = 4096$ into 11,008 (§5.3).

Rotary position embeddings instead of a learned table. No position vector is added to the input. Instead, queries and keys are rotated by an angle proportional to their position before the dot product, so attention scores depend on relative position only (Su et al., 2021, derived in §5.5). There's no table with a hard edge at $T$, and context windows can later be stretched (§12.7).

No biases. Every linear layer and norm drops its bias. PaLM reported that removing them increased training stability for large models (Chowdhery et al., 2022), and they were a negligible share of the parameters anyway.

An untied head. At 7B parameters the output matrix is 2 percent of the model, so the saving from tying no longer pays for forcing the input and output representations to be the same. LLaMA unties.

Grouped-query attention. Key-value heads are shared by groups of query heads, which shrinks the KV cache of §6.5 by the group factor with little loss of quality (Ainslie et al., 2023). LLaMA-2 used it for its 34B and 70B models, and LLaMA-3 for every size.

Bigger vocabularies and longer contexts. LLaMA's 32,000-token SentencePiece vocabulary grew to 128,256 in LLaMA-3, which shortens sequences, especially for code and non-English text (§2.7). Context grew from 2,048 to 4,096 to 8,192, then to 128K with a long-context training stage.

The recipe barely changed. LLaMA trains with AdamW, $\beta = (0.9, 0.95)$, weight decay 0.1, clipping at 1.0, 2,000 warmup steps, and a cosine decay to 10 percent of the peak, GPT-3's choices almost exactly.

The last line is the property that defines rotary embeddings. A query at position 5 and a key at position 2 score exactly as a query at 15 and a key at 12 do, because only the offset survives the rotation. And there's a parameter-count consequence of all these changes, which the next exercise makes exact.

def swiglu_hidden(d, multiple_of, ffn_dim_multiplier=None):
    raise NotImplementedError

def llama_params(V, d, L, d_ff, n_heads, n_kv_heads):
    raise NotImplementedError

import numpy as np

def llama_block_fwd(x, W, n_head, n_kv_head, theta=10000.0):
    raise NotImplementedError

The nanochat column is a snapshot of its code in early 2026, and it keeps changing as its "time to GPT-2" speedrun improves (it also has extras such as value embeddings mixed into some layers' values). Notice that it went back from SwiGLU to a plain (squared ReLU) MLP, and that it replaced Adam for the weight matrices with Muon, an optimizer that orthogonalizes each update (§8.6). Architecture choices are empirical, and they keep being revisited.

llm.c and nanochat

Karpathy's two follow-up projects are the best places to see a whole system end to end, and both are small enough to read.

llm.c (2024) trains GPT-2 in about 5,000 lines of plain C and CUDA, with no PyTorch at all, matching the PyTorch reference step for step. It reproduced GPT-2 124M (90 minutes, about $20, on 8 A100s) and then the full 1.6B model, on one node of 8 H100s in 24 hours for about $672, 32,000 steps of $2^{20}$ tokens each, about 33.6 billion tokens. Reading its kernels next to the PyTorch code is the fastest way to learn what each line of a training loop costs (§11.3).

nanochat (2025) goes the whole way from raw text to a chat model in one repository, with tokenizer training, pretraining, supervised fine-tuning (Chapter 14), reinforcement learning (§18.4), evaluation, and an inference engine with a KV cache. It is configured by one dial, the depth of the network, from which width, learning rates, and training horizon are derived to be compute-optimal. It reports validation loss as bits per byte (§6.6) and tracks a benchmark score called CORE. Its leaderboard measures the wall-clock time to match the original GPT-2's CORE score on one 8-GPU H100 node. The first entry, in January 2026, took 3.04 hours, and by March 2026 it was down to 1.65, with fp8 matmuls, a bigger batch, and a better pretraining dataset among the steps.

So where did the factor of a thousand in the puzzle come from? Nobody has an exact accounting, but the ingredients are clear and they multiply. The hardware delivers far more low-precision FLOPs per dollar. Software turns more of those FLOPs into useful work, through fused kernels, FlashAttention, bfloat16 and now FP8, and compiled graphs. Better-filtered data reaches the same score from a fraction of the tokens. And architecture and optimizer changes (rotary positions, better norms, Muon) make each token of training count for more. Nothing on that list is exotic. Each item is a section of this book.

Common bugs when scaling

Bugs that a 100K-parameter model on a laptop forgives become expensive at scale, because a run takes days before the symptom is unmistakable. Here are the ones that experienced people still hit, with the symptom that gives each away.

The meta-lesson is the one from §6.3, applied with more discipline. Before the big run, overfit one batch, check the loss at step zero against $\ln V$, compare against a known-good reference at small scale, and log the gradient norm, the learning rate, and the tokens seen from the very first step.

In PyTorch

Here is the LLaMA-style model in PyTorch. The meta device builds LLaMA-2 7B's shapes instantly, so we can check the parameter count against the published 6.74B, and then we check one block against the NumPy version above in float64.

The commands that run the real thing are short. Multi-GPU training with PyTorch launches one process per GPU with torchrun (§10.2 explains what DDP does underneath), and llm.c and nanochat each wrap a full run in one script.

# PyTorch, one node with 8 GPUs, data parallel (the build-nanogpt train script)
torchrun --standalone --nproc_per_node=8 train_gpt2.py

# llm.c, GPT-2 124M on 10B FineWeb tokens, 8 GPUs (flags as in the reproduction discussion)
mpirun -np 8 ./train_gpt2cu -i "dev/data/fineweb10B/fineweb_train_*.bin" -j "dev/data/fineweb10B/fineweb_val_*.bin" \
  -o log124M -e "d12" -b 64 -t 1024 -d 524288 -r 1 -z 1 -c 0.1 -l 0.0006 -q 0.0 -u 700 -n 5000 -v 250 -s 20000 -h 1

# nanochat, the whole pipeline from tokenizer to chat model
bash runs/speedrun.sh

Read more

• Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. Section 2 lists every architectural change from GPT-style models and the training recipe.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Grouped-query attention, longer context, and the GPU-hours used for each size.
• Brown et al. (2020). Language Models are Few-Shot Learners. The GPT-3 configurations and hyperparameters that GPT-2 reproductions borrow.
• Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. MFU defined, no biases for stability, and SwiGLU at scale.
• Karpathy. llm.c. GPT-2 training in C and CUDA, with the 124M and 1.6B reproductions documented in its discussions.
• Karpathy. nanochat. The full pipeline from tokenizer to chat model, and the time-to-GPT-2 leaderboard.
• Penedo et al. (2024). The FineWeb Datasets. The web dataset behind the modern GPT-2 reproductions.
• Shazeer (2020). GLU Variants Improve Transformer. SwiGLU and its relatives.

Watch next

▶

▶