Tensor parallelism hit a wall at the edge of the node. Llama 3 70B needs 1.13 TB of model states, and eight-way tensor parallelism leaves 141 GB on each GPU, still almost twice what an 80 GB GPU holds. We need to split the model further, across nodes, where the links are nine times slower. Tensor parallelism communicates activations at every layer, so it is out. What split needs almost no bandwidth?

Cut the model between layers. Put the first 20 layers on one group of GPUs, the next 20 on another, and so on. Each group, a stage, only has to send its output activations to the next stage and receive gradients back, once per layer group instead of four times per layer. That is pipeline parallelism. Its communication is tiny, and its problem is a different one. Stage 2 cannot start until stage 1 has finished, so most of the time most of the GPUs are waiting. This section derives exactly how much time is lost, called the bubble, and walks through the schedules that shrink it, from GPipe to 1F1B to interleaving to zero-bubble pipelines.

The naive pipeline, and why it wastes almost everything

Split a model of $L$ layers into $p$ stages of $L/p$ layers, one per device. Feed it one batch. Stage 1 runs its forward pass while stages 2 to $p$ wait. Then stage 2 runs while the others wait, and so on to stage $p$. The backward pass runs back up the same way. At every moment exactly one stage is busy, so each device is busy a fraction $1/p$ of the time. With eight stages, seven-eighths of the cluster is idle. We have gained memory (each device holds $1/p$ of the model) and lost all of the speed.

Micro-batches and the bubble, derived

GPipe (Huang et al. 2019) fixed this with the same idea as an assembly line. Split the batch into $m$ micro-batches. As soon as stage 1 finishes the forward pass of micro-batch 1, it hands the activations to stage 2 and starts micro-batch 2. Now several stages work at once, each on a different micro-batch. All $m$ forward passes flow down the pipeline, then all $m$ backward passes flow back up, and the gradients from all micro-batches are accumulated before a single optimizer step, so the result is exactly the same as processing the whole batch at once.

How much time is still wasted? Let each stage take $t_f$ for one micro-batch's forward pass and $t_b$ for its backward pass (about $2t_f$, §5.6). Follow the forward wave. Stage 1 starts micro-batch 1 at time 0. Stage $s$ can start micro-batch 1 only after the $s - 1$ stages before it have finished it, so it starts at $(s-1)t_f$, and the last stage starts at $(p-1)t_f$. The last stage then processes all $m$ micro-batches back to back, finishing the forward wave at $(p - 1 + m)\,t_f$. The backward wave is the mirror image, starting at the last stage and reaching the first after $(p - 1 + m)\,t_b$. So one step takes

Each stage does useful work for only $m(t_f + t_b)$ of that time, the same amount it would need with no pipeline at all. The rest, $(p-1)(t_f + t_b)$ per stage, is idle. The idle fraction is

In plain words, each stage waits $p - 1$ slots for the pipeline to fill and $p - 1$ slots for it to drain, out of $m + p - 1$ slots in total, so the waste shrinks as the number of micro-batches grows past the number of stages.

Narayanan et al. (2021) report the same quantity relative to the ideal time instead of the total, $(p-1)(t_f+t_b)/(m(t_f + t_b)) = (p-1)/m$. Both say the same thing, and you need $m \gg p$. With $p = 8$ stages and $m = 8$ micro-batches, the bubble is $7/15$, almost half. With $m = 64$ it is $7/71$, about 10%.

Here is a simulator that makes this concrete. A schedule is a list of operations per stage, in the order that stage runs them, and the simulator starts each one as soon as its stage is free and its inputs exist. A forward pass on stage $s$ needs the same micro-batch's forward on stage $s - 1$, and a backward pass needs the micro-batch's forward on its own stage and its backward on stage $s + 1$.

The simulator agrees with the formula exactly. Notice that the bubble does not depend on how fast the stages are, only on $p$ and $m$, as long as the stages are balanced. If one stage is slower than the others, every stage waits for it, and the slowest stage sets the pace of the whole pipeline.

def bubble_fraction(p, m, v=1):
    return (p - 1) / m

def min_microbatches(p, target, v=1):
    raise NotImplementedError

GPipe's memory problem, and 1F1B

The bubble argument says to use many micro-batches. GPipe's schedule makes that expensive. Every stage runs all $m$ forward passes before any backward pass, so it must keep the activations of all $m$ micro-batches at once. Activation memory per stage grows linearly with $m$, exactly the quantity we wanted to make large. GPipe itself paired the schedule with full recomputation, keeping only each stage's input per micro-batch.

The fix is to start backward passes as early as possible. In the one-forward-one-backward schedule (1F1B, from PipeDream by Harlap et al. 2018, and in the synchronous form "PipeDream-Flush" used by Narayanan et al. 2021), stage $s$ (counting from 0) runs $p - s - 1$ forward passes to fill the pipeline (the warm-up), then alternates one forward and one backward pass (the steady state), and finally runs the remaining backward passes (the cool-down). As soon as a micro-batch's backward pass has run on a stage, that stage frees its activations.

Count the micro-batches in flight on stage $s$, those whose forward has run there but whose backward has not. During warm-up it grows to $p - s$, and in the steady state every new forward is matched by a backward, so it stays there. The first stage holds at most $p$ micro-batches' activations, instead of $m$. And the bubble is unchanged. 1F1B rearranges the same work into the same total time, $(m + p - 1)(t_f + t_b)$, because the critical path, filling the pipeline and draining it, is the same.

There is a memory identity hiding here, noted by Korthikanti et al. (2022). The first stage holds $p$ micro-batches, each with the activations of $L/p$ layers, which is $L$ layers' worth of one micro-batch. So under 1F1B, the peak activation memory on the first stage equals that of the whole unsplit model on one micro-batch, whatever $p$ is. Pipeline parallelism splits the weights $p$ ways, but not the peak activations.

def make_schedule(p, m, kind):
    return [[("F", i) for i in range(m)] + [("B", i) for i in range(m)] for _ in range(p)]   # GPipe for both

def bubble(p, m, kind, tf=1.0, tb=2.0):
    raise NotImplementedError

Interleaving, more stages per device

The bubble is $p - 1$ slots of $t_f + t_b$ each. We cannot make $p - 1$ smaller without fewer stages, but we can make the slots shorter. Narayanan et al. (2021) give each device $v$ small chunks of the model instead of one large one. With $p = 4$ devices and $v = 2$, device 1 holds layers 1 to 10 and 41 to 50, device 2 holds 11 to 20 and 51 to 60, and so on, so a micro-batch travels around the devices twice. Each chunk has $1/v$ of the layers, so each forward and backward slot takes $t_f/v$ and $t_b/v$. The pipeline still needs $p - 1$ slots to fill and drain, but they are $v$ times shorter, so

In plain words, interleaving divides the bubble by the number of chunks per device.

It is not free. Every micro-batch now crosses $v$ times as many stage boundaries, so point-to-point communication grows by $v$, and the schedule is more intricate. Megatron's version requires $m$ to be a multiple of $p$ (Llama 3 modified the schedule to lift this restriction), and its peak activation memory grows slightly, by a factor of $1 + (p-1)/(pv)$ according to Korthikanti et al. The schedule processes micro-batches in groups of $p$, running chunk 0 for a group, then chunk 1 for the same group, and so on. The widget above includes it, and the cell below checks the formula with the same kind of simulator.

Zero-bubble pipelines

Qi et al. (2024) noticed that the backward pass is really two computations. For a linear layer $\mathbf{Y} = \mathbf{X}\mathbf{W}$, the input gradient $\partial\mathcal{L}/\partial\mathbf{X} = \mathbf{G}\mathbf{W}^\top$ is needed by the previous stage right away, but the weight gradient $\partial\mathcal{L}/\partial\mathbf{W} = \mathbf{X}^\top\mathbf{G}$ is needed only by the optimizer at the end of the step. Split the backward pass into $B$ (input gradients) and $W$ (weight gradients). Only $B$ is on the critical path between stages, so $W$ can be postponed to fill the bubbles.

Their schedule ZB-H1 follows 1F1B but lets stage $s$ hold back the weight gradients of its first $s$ micro-batches and run them later, in the cool-down where 1F1B leaves stages idle. With $t_F = t_B = t_W$, the bubble shrinks to a third of 1F1B's with the same peak memory. ZB-H2 runs more forward passes during warm-up to fill the start as well and reorders the tail, reaching essentially no bubble, at the cost of more memory and of a trick to avoid synchronizing before the optimizer step. The widget shows ZB-H1. DeepSeek-V3's DualPipe goes further, feeding micro-batches from both ends of the pipeline at once and overlapping the computation of one micro-batch with the expert-parallel all-to-all of another (§13.4).

Choosing the number of stages and micro-batches

The micro-batch count is not a free choice. With a global batch of $B$ sequences, $n_{\text{dp}}$ data-parallel replicas, and micro-batches of $b$ sequences, each pipeline gets $m = B/(n_{\text{dp}}\,b)$ micro-batches. More data parallelism means fewer micro-batches per pipeline and a bigger bubble, and a larger micro-batch means fewer of them too. Take a 4M-token batch of 8,192-token sequences, 512 sequences. With $n_{\text{dp}} = 16$ and $b = 1$, each pipeline gets $m = 32$. With $p = 8$ stages the bubble fraction is $7/39$, 18%. Interleaving with $v = 4$ brings it to $7/135$, about 5%.

Stages must also be balanced, because the slowest stage sets the pace. A Transformer's layers are identical, but the first stage also holds the embedding and the last holds the output layer and the loss, which for a large vocabulary is the cost of several layers. Llama 3 handled this by giving the first and last stages one Transformer layer fewer. In general, splitting a chain of layers with costs $c_1, \dots, c_L$ into $p$ contiguous stages that minimize the largest stage cost is the classic linear partition problem.

Pipelining also constrains how the data-parallel dimension shards its state. Every stage runs $m$ backward passes per optimizer step, so ZeRO-2 would have to reduce-scatter its gradient shards after each of them, and ZeRO-3 would also gather the stage's weights again for every micro-batch, multiplying the data-parallel traffic by about $m$ (§10.3). Pipelined runs therefore pair the pipeline with ZeRO-1 (DeepSeek-V3) or limit the sharding, as Llama 3 did by not resharding the weights after the forward pass, which removes the second gather of every micro-batch.

def balance_stages(costs, p):
    L = len(costs)
    sizes = [L // p] * p
    sizes[-1] += L - sum(sizes)
    return sizes                         # equal layer counts, ignoring the costs

What a pipeline communicates

Between two stages flows one activation tensor per micro-batch forward and one gradient of the same shape backward, $2bTd$ bytes each in 16 bits, sent point to point. For Llama 3 70B with $T = 8192$ and $b = 1$ that is 134 MB per direction per micro-batch. A stage of $L/p = 10$ layers ($p = 8$) does about $6 \times 70.6\times10^9/8 \times 8192 \approx 4.3\times10^{14}$ FLOPs of forward and backward work on that micro-batch, about a second at 400 TFLOP/s, while moving 268 MB over a 50 GB/s link takes about 5 milliseconds. The communication is half a percent of the compute, which is why pipeline parallelism is the strategy of choice across nodes. With interleaving, multiply the bytes by $v$, still small.

The real costs of pipelining are elsewhere. There is the bubble. There is the activation memory of the first stage, which 1F1B bounds but does not shrink. There is the constraint that $m$ be large, which fights with data parallelism for the global batch. And there is complexity, since every stage runs a different program and a schedule bug can deadlock the whole job.

In PyTorch

PyTorch's torch.distributed.pipelining package provides stages and schedules. You build the part of the model that belongs to this rank, wrap it in a PipelineStage, and choose a schedule. The first stage feeds inputs, the last computes the loss, and the middle ones only relay. This needs one process per stage and is shown for reference.

The model's forward must skip the parts set to None, so that a middle stage maps hidden states to hidden states. For interleaving, each rank builds several PipelineStage objects (one per chunk) and passes the list to ScheduleInterleaved1F1B.

Read more

• Huang et al. (2019). GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. Micro-batching, the bubble, and recomputation to bound memory.
• Harlap et al. (2018). PipeDream: Fast and Efficient Pipeline Parallel DNN Training. The one-forward-one-backward idea, first in an asynchronous form.
• Narayanan et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. PipeDream-Flush (1F1B), the interleaved schedule, and how pipeline, tensor, and data parallelism combine.
• Qi et al. (2024). Zero Bubble Pipeline Parallelism. Splitting the backward pass into B and W, and the ZB-H1 and ZB-H2 schedules.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Section 3.2 describes DualPipe, a bidirectional schedule that overlaps computation with expert communication.
• PyTorch. torch.distributed.pipelining. Stages, schedules, and the interleaved and zero-bubble implementations.
• Tazi et al. (2025). The Ultra-Scale Playbook. Its pipeline chapter walks through every schedule with diagrams and measured bubbles.

Watch next

▶

▶