Suppose someone hands you 8,000 H100 GPUs for two months and asks for the best base model you can build. Every section of this chapter is a decision you now have to make. Which data, filtered how, deduplicated how, mixed in what proportions (§8.2 to §8.4)? Which number formats (§8.5)? Which optimizer, schedule, and batch size (§8.6, §8.7)? Which stability measures, and how will you notice when something breaks (§8.8)? And before all of that, a question of arithmetic. How big a model, on how many tokens, fits in your budget at all?

This section works through that arithmetic and then takes apart three published runs that made these decisions at frontier scale, LLaMA 3 405B (Dubey et al. 2024), OLMo 2 (Team OLMo 2024), and DeepSeek-V3 (DeepSeek-AI 2024). They are worth studying side by side because they made different bets. LLaMA 3 is a dense model trained with conservative, well-understood choices at enormous scale. OLMo 2 is fully open, with every stability fix documented. DeepSeek-V3 is a sparse mixture of experts trained in FP8 for a small fraction of LLaMA 3's compute. It is also a preview. Its architecture rests on two ideas this book has not taught yet, multi-head latent attention (§12.5) and the mixture of experts (Chapter 13), so here we treat them as black boxes with known costs and study the training recipe around them. Chapters 12 and 13 open the boxes. We finish with the stages of a run, how it is evaluated while it trains, and what its operators watch.

The budget, in three formulas

Three numbers bound every plan, the compute, the time, and the memory.

Compute. Training a Transformer with $N$ parameters (not counting embeddings) on $D$ tokens costs about

2 FLOPs per parameter per token for the forward pass and 4 for the backward (§5.6 derives it, and §9.2 uses it for scaling laws). Attention adds a term that grows with the sequence length $T$, about $12LdT$ FLOPs per token for $L$ layers of width $d$ (Chowdhery et al. 2022), which is small at $T = 4{,}096$ for large models but not for long-context training. For a mixture of experts, $N$ is the number of active parameters per token.

Time. A GPU with peak throughput $F$ that achieves a model FLOPs utilization (MFU) of $u$ does $uF$ useful FLOPs per second, so

for $n$ GPUs. MFU, which you met in §6.7, counts only the FLOPs the model needs (the $6ND$ plus attention), not recomputation or communication, so it measures how well the whole system turns hardware into training. Large dense runs reach about 35 to 45 percent. How to measure it from tokens per second, and how it differs from hardware FLOPs utilization, is in §11.2 and §10.7.

Memory. Mixed-precision AdamW needs 16 bytes per parameter for weights, gradients, and optimizer state (§8.6), plus activations, which depend on batch size, sequence length, and how much is recomputed. Sharding across GPUs divides the 16 bytes (§10.3), and the full accounting, activations included, is in §10.1.

Let's check LLaMA 3 405B against its paper. With $N = 405\times10^9$ and $D = 15.6\times10^{12}$, $6ND = 3.79\times10^{25}$, matching the reported $3.8\times10^{25}$ FLOPs. At 400 TFLOP/s per GPU (41 percent of the H100's 989 dense BF16 TFLOP/s, their measured rate on 16,384 GPUs), that is $3.79\times10^{25}/(4\times10^{14}) \approx 9.5\times10^{10}$ GPU-seconds, or 26 million GPU-hours, and on 16,384 GPUs about 67 days of pure computation, before any interruption.

(OLMo 2's GPU count and MFU here are illustrative, chosen to show the arithmetic. The paper reports its hardware separately.) The DeepSeek-V3 line is worth a second look. Its 2.664 million H800 GPU-hours of pretraining for $3.3\times10^{24}$ active-parameter FLOPs imply an MFU of about 35 percent measured against the BF16 peak, or about 17 percent against the FP8 peak it actually used, and $6ND$ undercounts its work (attention, the multi-token prediction module, and the routing are extra). The headline is the ratio. LLaMA 3 405B used about 11.5 times more compute.

def plan_run(n_params, n_tokens, n_gpus, peak_flops, mfu, gpu_mem_bytes=80e9, bytes_per_param=16, active_params=None):
    raise NotImplementedError

LLaMA 3 405B, the conservative giant

Meta's paper is the most detailed public account of a dense frontier run. Its guiding principle, stated by the authors, was to manage complexity, choosing a standard dense Transformer over a mixture of experts and well-understood training methods over newer ones, and to win through data, scale, and engineering.

A few decisions deserve comment. The model size came from their own scaling laws, which predicted a compute-optimal model of 402B parameters on 16.55T tokens for this budget, so the 405B model is close to compute-optimal, while the 8B and 70B models were trained far past that point (the 8B saw about 1,900 tokens per parameter) because smaller models are cheaper to serve (§9.4). Attention used a mask that prevented tokens from attending across document boundaries within a packed sequence (§4.4), which mattered most for long sequences. And the long-context stages extended the context gradually, moving to the next length only when short-context benchmarks had recovered and a needle-in-a-haystack test passed at the current length (§12.7).

A batch ramp changes how token budgets turn into optimizer steps, and schedules are written in steps, so planning a run means converting between the two. LLaMA 3's ramp is a good test. Its batches were $2^{22}$, $2^{23}$, and $2^{24}$ tokens (1,024 sequences of 4,096, 1,024 of 8,192, and 2,048 of 8,192), switching after 252 million and 2.87 trillion tokens. That gives $61 + 342{,}101 + 758{,}768 \approx 1.10$ million steps for 15.6 trillion tokens, in line with the 1.2 million steps over which its cosine schedule was set to decay. Most of the tokens, and most of the steps, come at the largest batch. Running the whole budget at 16 million tokens per step would take about 930,000 steps, so the ramp adds about 170,000 optimizer steps (18 percent more steps for the same tokens), in exchange for not wasting tokens early, when the critical batch is small (§8.7).

import math
import numpy as np

def ramp_steps(boundaries, batch_tokens, total_tokens):
    return np.array([math.ceil(total_tokens / batch_tokens[-1])] + [0] * (len(batch_tokens) - 1))   # placeholder: one batch size throughout

def tokens_after(step, boundaries, batch_tokens, total_tokens):
    raise NotImplementedError

The paper is just as valuable for its account of operations. During a 54-day snapshot of pretraining, the job was interrupted 466 times, 47 times for planned maintenance and 419 times unexpectedly, about 78 percent of them from confirmed or suspected hardware problems, with faulty GPUs the largest single category. Only three interruptions needed significant manual intervention, because detection and restart were automated. At this scale, failure is the normal state, and the training system has to be designed around it.

What does that failure rate cost? §10.7 derives the checkpoint interval that minimizes wasted time, Young's $\tau^\star = \sqrt{2\delta M}$ for a checkpoint that stalls training for $\delta$ and a mean time between failures $M$, with a wasted fraction of about $\sqrt{2\delta/M}$. With LLaMA 3's 419 unexpected failures in 54 days, one every 3.1 hours, and a checkpoint that stalls training for a minute, that is a checkpoint every 19 minutes and still about 10 percent of the time lost to checkpointing and rework. That is why fast, asynchronous checkpointing and quick failure detection are among the most valuable engineering investments in a large run.

OLMo 2, the open recipe

Ai2's OLMo 2 matters less for its scale than for its openness. The data, code, intermediate checkpoints, and logs are all public, and the paper documents what went wrong in earlier versions and how each fix was chosen. Its 7B model trained on 4.05 trillion tokens in total, and the 13B on 5.6 trillion.

The run has two stages. Stage 1 trained on OLMo-Mix-1124, 3.9 trillion tokens of mostly web text (DCLM-Baseline at its core) plus code, papers, and math. Stage 2, mid-training, trained on Dolmino-Mix-1124, a smaller mix of high-quality web text, curated math, and instruction-like data, while the learning rate decayed linearly to zero. For the 7B model, mid-training ran three times on 50 billion tokens with different data orders, and the three results were averaged into one model. The 13B ran three times on 100 billion tokens and once on 300 billion, and all four were averaged. Mid-training was 5 to 10 percent of the total compute and accounted for much of the final model's gains on math and knowledge benchmarks. (This is the annealing of §8.4, made into a stage of its own.)

The stability changes are a catalogue of §8.8, and the paper shows each one's effect on loss and gradient-norm spikes.

• Reordered norms. Each block normalizes the output of attention and of the MLP before adding it to the residual stream, $\mathbf{h} = \mathbf{x} + \operatorname{RMSNorm}(\operatorname{Attn}(\mathbf{x}))$, instead of normalizing the input.
• QK-norm, with RMSNorm on queries and keys.
• z-loss on the output logits.
• No weight decay on embeddings, and every parameter initialized from a normal distribution with standard deviation 0.02.
• A smaller Adam epsilon, $10^{-8}$ instead of $10^{-5}$.
• Data filters that remove documents with long runs of repeated $n$-grams.

The paper measures each change with a gradient-norm spike score, the percentage of steps at least 7 standard deviations from the rolling mean of the previous 1,000 (Exercise 8.8d in §8.8). In an ablation built to spike quickly (with a shortened warmup), the new initialization alone took the score from 0.40 to 0.03, and reordered norms together with QK-norm took another run's score from 0.108 to 0.069. The 7B model used a peak learning rate of $3\times10^{-4}$, 2,000 warmup steps, a cosine schedule (planned to reach 10 percent of the peak at 5 trillion tokens, then cut short at 4 trillion), batches of about 4 million tokens, and gradient clipping at 1.0.

DeepSeek-V3, the efficient sparse model

This part is a preview of Chapters 12 and 13, and you can read it with two facts in hand. A mixture-of-experts layer replaces one feed-forward network with many, of which a router picks a few per token, so the model has far more parameters than any one token uses. And multi-head latent attention stores a small compressed vector per token in the KV cache instead of full keys and values. DeepSeek-V3 is a mixture-of-experts model with 671 billion total parameters, of which 37 billion are active for each token (§13.5). It has 61 layers of width 7,168, with multi-head latent attention to shrink the KV cache (§12.5). Every feed-forward layer except the first three is a mixture of one shared expert and 256 routed experts, of which each token uses 8, with routing limited to at most 4 nodes per token to bound communication. Load is balanced mainly by an auxiliary-loss-free bias on the router, with only a tiny sequence-wise balance loss as a safeguard (§13.3). It also predicts one extra future token with the sequential multi-token prediction module of §8.1, which adds training signal and can be used for speculative decoding.

Two remarks. The schedule is essentially warmup-stable-decay (§8.7), with a long constant phase and two low-rate stages at the end. And the paper reports no irrecoverable loss spikes and no rollbacks during the entire run, which, given PaLM's and OPT's experiences, is itself a result. The $5.6 million figure needs its footnote, which the authors give. It counts only the GPU-hours of the official run, not the research, ablations, and failed experiments that led to it, nor the cost of the cluster itself. It is still a striking demonstration of how much efficiency is available from sparsity, FP8, and engineering for communication (§13.4).

Three runs side by side

The comparison shows how much the recipes agree. All three use AdamW (DeepSeek-V3 reports $\beta_2 = 0.95$, and both it and OLMo 2 clip at 1.0, while the LLaMA 3 paper does not list its betas or clipping threshold), warm up for thousands of steps, decay to a small learning rate at the end with a shift toward better data, ramp or fix large batches, and extend context in a late stage. The differences are in the bets. LLaMA 3 bet on scale and simplicity, OLMo 2 on careful stability engineering in the open, DeepSeek-V3 on sparsity and low precision.

The stages of a run

A modern pretraining run is a sequence of phases, each with its own data mixture, learning rate, and sequence length.

1. Main pretraining. Most of the tokens, at a short context (4K to 8K), on the broad mixture, with the learning rate at or near its peak.
2. Mid-training, or annealing. A shift to the best data, more math, code, and instruction-like text, while the learning rate decays (LLaMA 3's last 40 million tokens, OLMo 2's 50 to 300 billion).
3. Long-context extension. Training on long documents with a larger RoPE base or a method such as YaRN, extending the context step by step (LLaMA 3's six stages and about 800 billion tokens, DeepSeek-V3's two short stages). The order of stages 2 and 3 varies between labs.
4. Checkpoint averaging. Averaging several late checkpoints or several annealing runs (Polyak averaging, model souping) gives a slightly better and more robust final model at no training cost.
5. Hand-off to post-training, where supervised fine-tuning and preference optimization turn the base model into an assistant (§14.1).

Evaluating a run while it trains

You cannot wait until the end to find out whether a run is good. Teams evaluate checkpoints continuously, and this is harder than it sounds, because early checkpoints score near chance on most benchmarks.

Held-out loss is the most sensitive signal. Track it on a fixed validation set per domain (web, code, math, each language), measured in a tokenizer-independent unit such as bits per byte when comparing models with different tokenizers (§6.6). A domain whose loss rises while the others fall points at a mixture or data problem.

Benchmarks chosen for early signal. FineWeb's team selected tasks on which small models trained on a few billion tokens already score clearly above chance and rank datasets consistently, and DCLM's CORE suite was built the same way. For multiple-choice tasks, scoring the likelihood of each answer as a continuation (the cloze formulation) gives signal much earlier than asking the model to output a letter, which small models cannot do reliably.

Predicting the end. Because loss falls sharply during the final decay (§8.7), a mid-run checkpoint underestimates the final model. Labs either anneal a copy of a mid-run checkpoint to see where it would land, which is cheap with a warmup-stable-decay schedule, or compare the loss curve with a scaling-law prediction made from small runs before the big one started (§9.1).

Decontamination and honesty. Every evaluation set should be checked against the training data (§8.3), and a team that picks its data and checkpoints by the same benchmarks it reports will overestimate its model. Keep a held-out set of evaluations that nobody optimizes against. How to evaluate models well is a field in its own right (benchmark design, statistics, and contamination).

What the operators watch

Combining the stability dashboard of §8.8 with the systems view, a large run is watched on three levels.

The best single habit is to compare every curve with a prediction made before the run started. A loss that is slightly worse than predicted is the earliest warning that a data or optimization choice did not transfer from the small-scale experiments, and it shows up long before any benchmark does.

In PyTorch

The $6N$ rule is easy to check against PyTorch's own FLOP counter. Here we count the FLOPs of one forward and backward pass through a small GPT-style model and compare them with $6N$ per token plus the attention term $12LdT$.

The counter agrees with the formula exactly, because every FLOP of a Transformer's training step is in its matrix products, and the formula counts all of them. (The counter, like the formula, counts the full $T\times T$ attention product even though the causal mask makes half of it unnecessary, which is the convention MFU calculations use.)

Read more

• Dubey et al. (2024). The Llama 3 Herd of Models. Sections 3 and 4 are the most complete public account of a frontier pretraining run, infrastructure and failures included.
• Team OLMo (2024). 2 OLMo 2 Furious. A fully open recipe with the stability ablations and the mid-training stage.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. MoE architecture, FP8 training, and the cost accounting.
• Chowdhery et al. (2022). PaLM. Defines model FLOPs utilization and documents training at 540B on TPUs.
• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. The token-parameter trade-off behind every budget plan, studied in depth in Chapter 9.
• Young (1974). A first order approximation to the optimum checkpoint interval. The checkpoint formula, still the starting point.

Watch next

▶

▶

▶