Here is a question that comes up, in one form or another, in almost every research-engineering interview at a lab. You have 1,024 H100s and a 70B-parameter model to train on 1.4 trillion tokens. How do you lay out the job, how long will it take, and what will go wrong? A good answer is not a list of acronyms. It is a chain of numbers, how much memory each GPU needs under each layout, how much time each strategy spends communicating, how much the pipeline idles, what fraction of the hardware's peak you can expect, and how often the job will crash.

Every piece of that chain is in the previous six sections, except two. We need the true cost of activation checkpointing and the arithmetic of failures and checkpoints. Efficiency is measured with model FLOPs utilization (MFU), which §6.7 defined, and we recap it only to read it off a layout. This section adds the missing pieces, then puts everything together into a worked plan.

Activation checkpointing and its 33 percent

Full activation checkpointing (or recomputation) keeps only each layer's input during the forward pass, $2Tbd$ bytes per layer in 16 bits, and throws the rest away. In the backward pass, just before differentiating a layer, it reruns that layer's forward pass from the saved input to rebuild the activations, then runs the backward pass as usual and frees them again. Only one layer's full activations exist at a time, on top of the saved inputs.

Count the compute. Recall from §5.6 that the backward pass costs about twice the forward pass, because each matrix multiply in the forward pass has two in the backward pass (one for the input gradient, one for the weight gradient). So a training step without recomputation costs $1 + 2 = 3$ forward passes, and with full recomputation $1 + 1 + 2 = 4$.

In plain words, full recomputation adds one extra forward pass to the three a step already costs, 33% more compute, and so at the same hardware efficiency it cuts throughput to three quarters.

Korthikanti et al. (2022) measured an overhead of 30% to 40% in practice, consistent with this count. In FLOPs per token, with $N$ the parameters taking part in matrix multiplies, $L$ layers of width $d$, and sequences of $T$ tokens, training costs $6N + 12LdT$ without recomputation and $8N + 16LdT$ with it.

Which attention term is that? The $12LdT$ is PaLM's convention (§5.6). It counts the full $T\times T$ score matrix, $4Td$ forward FLOPs per token per layer for $\mathbf{Q}\mathbf{K}^\top$ and the weighted values, tripled for training. A causal model needs only the scores on and below the diagonal, and a kernel that skips the masked half, as FlashAttention's causal mode does, executes about half of that, $6LdT$, the causal count of §4.7. This section uses the full $12LdT$ throughout, because MFU is conventionally reported that way and because it is what the published numbers we reproduce below assume. Whenever you quote a FLOP count, say which one you mean.

There are cheaper points on the same curve. Checkpointing every $k$ layers keeps one input per $k$ layers and recomputes $k$ layers at a time, so memory is $L/k$ saved inputs plus $k$ layers of full activations during the recomputation. Choosing $k = \sqrt{L}$ minimizes the sum, which is Chen et al.'s (2016) $O(\sqrt L)$ memory for one extra forward pass. Selective recomputation (Korthikanti et al.) recomputes only the attention core ($\mathbf{Q}\mathbf{K}^\top$, softmax, dropout, and the weighted values), whose activations are large ($5hT^2b$ bytes) but whose FLOPs are a small share of the layer. For GPT-3 they report a FLOPs overhead of 2.7%, which matches the attention core's share of the forward pass, $4T/(24d + 4T)$ at $d = 12{,}288$ and $T = 2048$ (relative to the whole training step it is a third of that). FlashAttention gets the same memory saving inside the attention kernel.

The rule is to recompute only as much as you must. Llama 3 trained at 8,192 tokens without any activation checkpointing, by choosing a layout in which the activations fit.

PyTorch implements checkpointing as a wrapper around a function or module. On a CPU we can count what it saves and how many forward passes it runs.

With checkpointing, each block keeps only its 8 KB input instead of 40 KB, and every block's forward pass runs twice.

def train_flops_per_token(N, L, d, T, recompute="none"):
    return 6 * N

def recompute_slowdown(N, L, d, T, recompute):
    return 1.0

MFU, read off a layout

How efficient is a training run? The number labs report is model FLOPs utilization, defined by Chowdhery et al. (2022) in the PaLM paper and introduced in §6.7. It is the throughput you achieved, counted in the FLOPs the model needs, divided by the peak FLOPs of the hardware,

where $N$ is the number of parameters, $L$ the layers, $H$ the attention heads, $Q$ the head dimension ($HQ = d$ for standard multi-head attention), $T$ the sequence length, and $F_{\text{peak}}$ the peak dense throughput of one accelerator.

In plain words, MFU asks what fraction of the hardware's arithmetic went into computing the model's forward and backward passes, counting each necessary FLOP once.

§6.7 lists what keeps an MFU honest (model FLOPs only, the dense peak, whole steps by the wall clock), and §11.2 adds the hardware view, hardware FLOPs utilization (HFU), which also counts the recomputation of the previous subsection and so can exceed MFU by up to $4/3$ (PaLM 540B reported an MFU of 46.2% and an HFU of 57.8%). Two conventions matter when you compare published runs. $N$ should count the parameters that take part in matrix multiplies (the $N_{\text{mm}}$ of §5.6, since an untied input embedding is a lookup), and the attention term should say whether it counts the full score matrix, as PaLM's $12LHQT$ does, or only the causal half, $6LHQT$. The cell reproduces three published MFUs from the throughputs their authors reported.

These reproduce the published numbers, 45.7% and 46.2% for PaLM and 29.8% and 30.3% for MT-NLG (the appendix truncates the last two to 29.7% and 30.2%). For the three Llama 3 405B configurations the division gives 43.5%, 40.4%, and 38.4%, against the reported 43%, 41%, and 38% (the per-GPU throughputs in the report are themselves rounded). For orientation, dense pretraining at scale on modern GPUs typically reports MFU between about 35% and 55%. The rest goes to communication that is not hidden, pipeline bubbles, memory-bound operations such as norms and softmaxes (§11.2), and kernels that do not reach peak even when compute-bound. The first two are set by the layout, which is why the same model on the same GPUs can differ by several points of MFU between layouts (six in the worked plan below, between a plain and a pipelined layout).

Memory per GPU under a full layout

Now combine every memory formula of the chapter into one. A layout has tensor-parallel degree $t$ (with or without sequence parallelism), pipeline degree $p$ (with $v$ interleaved chunks per device), and data-parallel degree $n_{\text{dp}}$ with a ZeRO stage, and $n_{\text{gpu}} = t\,p\,n_{\text{dp}}$. Each GPU owns $N/(tp)$ parameters, whose 16 bytes of model states are then divided by ZeRO as in §10.3. Activations per layer follow Korthikanti et al.'s table (§10.4). Under 1F1B the first pipeline stage stores $p$ micro-batches of $L/p$ layers, which is $L$ layers' worth of one micro-batch (§10.5), and interleaving multiplies that by $1 + (p-1)/(pv)$. So

where $a_{\text{layer}}$ is the per-layer activation bytes of Korthikanti et al.'s table for the given tensor degree, sequence parallelism, and recomputation, the subscript means that the three state terms are divided by $n_{\text{dp}}$ from ZeRO stage 3, 2, and 1 onward, respectively, and the interleaving factor is 1 when $v = 1$.

In plain words, tensor and pipeline parallelism divide the model states, ZeRO divides them further, tensor and sequence parallelism and recomputation shrink the activations, and pipelining leaves the activations of the first stage the same as the whole model's.

This is the formula behind the memory widget of §10.1 and the planner below. Real frameworks add a few gigabytes of communication buffers and fragmentation, so leave 10% to 20% of headroom.

def per_gpu_memory(N, L, d, h, T, b=1, t=1, p=1, n_dp=1, zero=0, sp=False, recompute="none", v=1):
    states = 16 * N / (t * p * n_dp)
    acts = L * 34 * T * b * d / t
    return {"states": states, "activations": acts, "total": states + acts}

Choosing a 3D or 4D layout

With memory, communication, and bubbles all quantified, the layout follows from a short sequence of decisions. This is the recipe Narayanan et al. (2021) arrived at for Megatron, refined by Llama 3's experience.

1. Use tensor parallelism inside the node, and only as much as you need. Its traffic is on the critical path and scales with the activations (§10.4), so it lives on NVLink, usually $t = 8$ for large models and less for small ones, where its smaller matrices cost more efficiency than they save. Always pair it with sequence parallelism.
2. Add pipeline stages until the model states and activations fit. Pipelining crosses nodes cheaply (§10.5). Use the fewest stages that fit, because every stage adds bubble, and use interleaving to shrink the bubble.
3. Add context parallelism when sequences are long, so that each GPU's share of tokens keeps activations bounded (§10.6). Llama 3 used none at 8K tokens and 16-way at 128K.
4. Spend everything else on data parallelism, with ZeRO-1 or FSDP to shard what remains. Check that the global batch gives each pipeline enough micro-batches ($m \ge p$, ideally $m \gg p$) and each GPU enough tokens to hide the data-parallel traffic (§10.2 and §10.3).
5. Use the smallest micro-batch that keeps the GPU efficient, often 1 or 2 sequences, and recompute only if nothing else fits.
6. Order the mesh by bandwidth need, tensor innermost, then context, pipeline, and data outermost.

The model-parallel size $t \cdot p$ is the key quantity. Making it as small as memory allows keeps the expensive communication small and the data-parallel degree large.

How published runs were laid out

Two frontier runs report their layouts in enough detail to check the recipe against. Llama 3 405B (Dubey et al. 2024, Table 4) is dense, and DeepSeek-V3 (DeepSeek-AI 2024) is a mixture of experts with 671B parameters, 37B of them active per token.

Llama 3 follows the recipe step by step. Tensor parallelism fills the 8-GPU node. A 405B model's states are $16 \times 405.9\times10^9 \approx 6.5$ TB. Splitting the model over the $8 \times 16 = 128$ GPUs of one replica leaves about 51 GB of states per GPU before any data-parallel sharding, and only $2N/(tp) \approx 6.3$ GB of bfloat16 weights, which Llama 3 keeps gathered from the forward pass to the backward pass while FSDP shards the gradients and optimizer state over the data-parallel ranks. Everything else is data parallelism, and the global batch then fixes how many micro-batches each pipeline gets. The global batch stayed at about 16M tokens, 2,048 sequences of 8,192, so doubling the GPUs from 8,192 to 16,384 doubled the data-parallel degree and halved each pipeline's micro-batches, from 32 to 16 sequences per replica, and MFU fell from 43% to 41%, as a larger bubble predicts. For 128K-token documents, 16-way context parallelism was carved out of the data dimension, and the same 16M-token batch became 128 sequences, 16 per replica.

DeepSeek-V3 makes the opposite choice on tensor parallelism and uses none. Its report credits memory savings for that, among them recomputing the RMSNorms and the up-projections of its multi-head latent attention (§12.5) in the backward pass and keeping an exponential moving average of the weights in CPU memory. Its experts, 256 routed ones per MoE layer, are spread by 64-way expert parallelism instead, so the per-layer communication that matters is the expert all-to-all (§13.4), not tensor-parallel all-reduces. Sixteen pipeline stages scheduled with DualPipe hide that all-to-all behind computation, and ZeRO-1 shards the optimizer state over the 128 data-parallel ranks. Both runs keep the model-parallel group as small as memory allows and spend the rest of the cluster on data parallelism.

def smallest_layout(N, L, d, h, T, n_gpus, mem_bytes, global_batch, gpus_per_node=8, b=1):
    return (gpus_per_node, 1, n_gpus // gpus_per_node)          # always TP = node size

Fault tolerance and checkpointing at scale

A single GPU fails rarely. Sixteen thousand of them fail all the time. During a 54-day stretch of Llama 3 405B's pretraining on 16,384 H100s, the job was interrupted 466 times, 419 of them unexpectedly, about once every three hours. About 78% of the unexpected interruptions were attributed to confirmed or suspected hardware issues, and GPU problems (faulty GPUs and HBM memory above all) accounted for 58.7%. Even so, the team kept the effective training time, the fraction of wall-clock time spent on useful training, above 90%. Two facts drive the engineering.

Failures scale with the cluster. If each GPU, with its share of the host, network, and cooling, fails independently at some rate, the job's failure rate is that rate times the number of GPUs, and synchronous training stops when any one of them fails. The mean time between failures (MTBF) of the job is the per-GPU MTBF divided by $n_{\text{gpu}}$. From the Llama 3 numbers, 419 failures over $54 \times 24 \times 16{,}384$ GPU-hours is one failure per 50,700 GPU-hours, so a 1,024-GPU job can expect one about every 50 hours, and a 100,000-GPU job one every half hour.

Checkpoints are large. A checkpoint must hold everything needed to resume exactly, the float32 master weights and Adam moments (12 bytes per parameter, so 847 GB for a 70B model), usually the bfloat16 weights, the data loader's position, random number generator states, and the learning-rate schedule's step. Every rank writes its own shard in parallel (PyTorch's distributed checkpointing does this), and asynchronous checkpointing first copies the shards to CPU memory, a fraction of a second, then writes them to storage in the background while training continues. Some systems also keep recent checkpoints in the CPU memory of peer nodes, so that recovery does not need to read from remote storage at all (Wang et al. 2023).

How often should you checkpoint? Each checkpoint costs $\delta$ seconds of stalled training. Each failure loses the work since the last checkpoint, on average half an interval, plus a restart time $R$. With an interval $\tau$ between checkpoints and a job MTBF of $M$, the fraction of time wasted is about

The first term falls with $\tau$ and the second grows, and setting $W'(\tau) = -\delta/\tau^2 + 1/(2M) = 0$ gives the classic result of Young (1974) and Daly (2006),

In plain words, checkpoint at the geometric mean of the checkpoint cost and the time between failures, and the waste then grows only like the square root of the checkpoint cost.

The approximation assumes $\delta \ll \tau \ll M$, and Daly (2006) derives the higher-order corrections that matter when the checkpoint cost is not small next to the MTBF. It explains why fast, asynchronous checkpointing pays off so much. Cutting $\delta$ from a minute to a second cuts the optimal waste from checkpoints and lost work by a factor of almost eight, leaving the restart time as the dominant cost, which is why labs also invest in fast restarts, hot spare nodes, and automatic detection and removal of bad hosts.

At 100,000 GPUs, even with asynchronous checkpoints, a 10-minute restart every half hour wastes about a third of the run, which is why frontier-scale training increasingly needs ways to continue through failures without a full restart, for example by swapping in spare nodes or dropping a failed data-parallel replica.

import numpy as np

def young_daly(delta, mtbf):
    return mtbf / 2

def wasted_fraction(tau, delta, mtbf, restart):
    raise NotImplementedError

Failures are not always loud. Stragglers, GPUs that run slowly because of thermal throttling or a flaky link, slow every synchronous step, so production systems watch per-rank step times. Hangs, where a collective waits forever because one rank crashed or took a different code path, are detected with timeouts on collectives and diagnosed with records of recent collective calls (Llama 3 relied on PyTorch's NCCL flight recorder). And silent data corruption, a GPU that computes wrong numbers without an error, shows up as loss spikes or as disagreement between replicas that should be identical, which is one more reason to monitor gradient norms and to be able to roll back to an earlier checkpoint (§8.8).

A worked plan, 70B on 1,024 H100s

Now the interview question. The model is Llama 3 70B's architecture ($L = 80$, $d = 8192$, 64 query heads and 8 key-value heads, SwiGLU width 28,672, vocabulary 128,256, $N = 70.6$ billion). The data is 1.41 trillion tokens, Chinchilla's 20 tokens per parameter (§9.3), in sequences of $T = 8192$. The cluster is 128 nodes of 8 H100s (80 GB, 450 GB/s NVLink per GPU, one 50 GB/s InfiniBand NIC per GPU). The global batch is 4M tokens, 512 sequences (§8.7).

The budget. Training costs $6N + 12LdT = 4.24\times10^{11} + 0.64\times10^{11} = 4.88\times10^{11}$ FLOPs per token, so $6.9\times10^{23}$ FLOPs in total. The attention term, counted the PaLM way, adds 15% to the $6N$ at this length and is 13% of the total. A causal kernel actually executes half of it, $6LdT$, so the work done is about 7% less than this count, and the MFU we report uses the full count, as published MFUs do. (This also uses all 70.6B parameters. Leaving out the 1.05B of the input embedding, as the matrix-multiply convention above recommends, lowers the total by 1.3%, which does not change the plan.) At 40% MFU, 1,024 GPUs deliver $1024 \times 989\times10^{12} \times 0.4 = 4.05\times10^{17}$ FLOP/s, so the run takes $1.7\times10^6$ seconds, about 20 days, and about 337,000 optimizer steps of 4.2M tokens.

The layout. Follow the recipe. Model states are 1.13 TB, so model parallelism is unavoidable. Tensor parallelism of 8, one node, splits the 64 query heads into 8 per GPU and the 8 key-value heads into 1 per GPU. With sequence parallelism and FlashAttention, one sequence's activations are $80 \times 34 \times 8192 \times 8192/8 = 22.8$ GB per GPU. That leaves the model states. The cell compares candidate layouts under a simple performance model. Matrix multiplies run at 55% of peak, tensor-parallel all-reduces are fully exposed on NVLink, data-parallel traffic over InfiniBand is hidden when it is shorter than the compute, and pipelines pay their bubble.

The first row confirms that tensor parallelism alone does not fit. The model states alone are 141 GB per GPU. Everything else fits, and the choice is about speed and simplicity.

• Tensor 8 with ZeRO-1 over 128 data-parallel replicas fits at about 59 GB, has no bubble, and its 0.7 seconds of data-parallel traffic hide easily behind roughly 2.6 seconds of backward compute. It is the simplest plan with the best estimate, about 45% MFU. ZeRO-2 would bring memory to about 41 GB at no extra communication, buying headroom for buffers.
• Tensor 8 with FSDP fits with the most headroom (24 GB), at the price of about 3 seconds of data-parallel traffic per step, which still just hides. It buys that headroom with the network. The weight gathers repeat for every micro-batch, so this plan is the most sensitive to a slower interconnect or to imperfect overlap.
• Adding a pipeline cuts memory further but costs a 16% bubble at $p = 4$ without interleaving. Interleaving recovers most of it. A pipeline becomes the right choice when memory is tight, for example with larger micro-batches, 40 GB GPUs, or a larger model.

So a defensible plan is tensor parallelism of 8 with sequence parallelism inside each node, ZeRO-1 or ZeRO-2 data parallelism across the 128 nodes (mesh order tensor, then data), FlashAttention with no other recomputation, 4 micro-batches of one sequence per replica, and asynchronous sharded checkpoints. The estimate is a step of about 4.5 seconds and roughly 45% MFU. Treat it as an upper bound, since it ignores memory-bound kernels and imperfect overlap, and a realistic outcome is 35% to 45%, so 20 to 25 days.

For the long-context stage at 128K tokens, keep the tokens per GPU the same by adding context parallelism. With $n_{\text{cp}} = 16$ and $t = 8$, each GPU holds 8,192 tokens, the same 22.8 GB of activations as before, and the data-parallel degree falls to 8. A 4M-token batch is 32 sequences, 4 per replica. One thing must change with it. ZeRO-1 over only 8 replicas would leave $4 \times 8.8 + 12 \times 8.8/8 \approx 49$ GB of states per GPU, 71 GB with the activations, uncomfortably close to 80. But the 16 context-parallel ranks of a replica hold identical weights and see identical gradients after their reduction, so the optimizer state can be sharded over all $16 \times 8 = 128$ ranks that hold the same tensor shard, which brings the states back to 36 GB. Megatron's distributed optimizer and the planner below shard this way.

Checkpointing and failures. At Llama 3's failure rate, 1,024 GPUs fail about every 50 hours, so expect about ten interruptions over the run. A checkpoint is about 1 TB, under 1 GB per GPU when sharded. With asynchronous saves costing about a second of stall, Young and Daly give an interval of about 10 minutes and, with 10-minute restarts, about 0.7% of time wasted. Synchronous saves of a minute would call for an interval of about 80 minutes and waste about 3%.

What to watch. Before the run, measure collective bandwidth with nccl-tests, profile one step to check that the data-parallel traffic really overlaps, and run a short job at full scale to find bad nodes. During it, track tokens per second and MFU, per-rank step times (stragglers), loss and gradient norm (instability or silent corruption, §8.8), and memory headroom.

In PyTorch

Production frameworks expose these choices as configuration. Here are the two pieces most often written by hand, selective activation checkpointing and distributed, asynchronous checkpoints. Both need GPUs and a process group and are shown for reference.

Distributed checkpoints can be loaded into a different layout than the one that saved them (more or fewer GPUs, different sharding), because they store each tensor's global shape and the offsets of its shards. That is what lets a job resume on a smaller cluster after losing nodes.

Read more

• Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. Appendix B defines MFU and HFU and computes them for PaLM and earlier models.
• Narayanan et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. How to combine tensor, pipeline, and data parallelism, with the takeaways the recipe above follows.
• Korthikanti et al. (2022). Reducing Activation Recomputation in Large Transformer Models. Selective recomputation, the cost of full recomputation, and memory under every layout.
• Llama Team, AI at Meta (2024). The Llama 3 Herd of Models. Section 3.3 covers 4D parallelism, MFU at 16K GPUs, and reliability in production.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Section 3.2 gives the layout of a frontier MoE run, with pipeline, expert, and ZeRO-1 data parallelism and no tensor parallelism.
• Young (1974). A first order approximation to the optimum checkpoint interval. The one-page derivation of $\sqrt{2\delta M}$.
• Daly (2006). A higher order estimate of the optimum checkpoint interval for restart dumps. The checkpoint interval derived above, with corrections beyond first order.
• Tazi et al. (2025). The Ultra-Scale Playbook. Thousands of benchmarked layouts and the lessons drawn from them.

Watch next

▶

▶

▶