A video generator turns a prompt into a short clip of moving pictures. It is an image generator with one more axis, time, and that axis changes almost everything. The frames must agree with one another, so they are generated together rather than one at a time. And a clip holds so many numbers that cost, more than quality, decides what can be built. This section shows how the image machinery of this chapter stretches to video, and counts what it costs at every stage.

Here is the problem in numbers. A five-second clip at 16 frames per second has $1 + 5\cdot16 = 81$ frames, a first frame and 80 more. At $832\times480$ pixels with three colors each, that is $81\cdot480\cdot832\cdot3 = 97{,}044{,}480$ numbers. Suppose you give each frame to a good image generator, with the same prompt and the same random seed. You get 81 good pictures and a bad film. Textures shimmer from frame to frame, a face changes identity every few frames, and nothing moves the way real things move. Each frame is plausible on its own, and together they are impossible.

So a video model samples all the frames jointly, as one object. We build one from the pieces of this chapter, in order. First comes an autoencoder that compresses time as well as space, then a count of the tokens that compression leaves, then the point at which attention over those tokens becomes the main cost. With those three pieces we follow one prompt all the way to frames, with every shape and every FLOP. Then come the attention patterns that trade quality against cost, one of which lets a model stream frames out the way a language model streams words. After them come the recipe the large systems share, conditioning on an image, keeping a clip consistent, and generating in real time. Every system number is checked against its paper or recomputed in a cell.

What a time axis adds

§29.1 measured two facts about real video. Consecutive frames are nearly identical, which is why video codecs compress by factors of hundreds. And the part that does change, the motion, is exactly what a viewer notices. Three consequences follow for a generator.

• Redundancy. Neighboring frames repeat each other, so the autoencoder should compress time as well as space. Then the denoiser never has to process the repetition.
• Joint modeling. Identity and motion are relations between frames. So every token must be able to exchange information with tokens at other times, in every layer or at least within a few layers.
• Cost. Even after compression a clip is tens of thousands of tokens, and the cost of attention grows with the square of that count.

A tiny clip makes all three concrete.

Compressing time with a causal 3D autoencoder

Recall the idea of §34.1. An autoencoder has an encoder $\mathcal{E}$ that shrinks an image to a small grid of latent vectors and a decoder $\mathcal{D}$ that maps the grid back to pixels. The encoder shrinks each side by a factor $f_s$ and keeps $c$ numbers, the channels, at each latent position. Diffusion then runs on the small latent instead of the pixels. For video there are two ways to build such an autoencoder.

The first reuses an image autoencoder on every frame. Video LDM (Blattmann et al. 2023a) and Stable Video Diffusion (Blattmann et al. 2023b) did this with the Stable Diffusion autoencoder, which has $f_s = 8$ and $c = 4$. It costs nothing new. But it leaves all the repetition between frames in the latent. And decoding each frame on its own causes flicker, because nothing ties the fine detail of one frame to that of the next. Video LDM had to add temporal layers to the decoder and fine-tune it on video to suppress the flicker.

The second way compresses time too. It adds temporal convolutions to the autoencoder, small filters that slide along the time axis at each pixel and mix a few neighboring frames. Almost every system since 2024 takes this route. The interesting design question is what such a filter does at the start of the clip, where there are no earlier frames to mix.

Here is the answer on a small case. A filter of length $k = 3$ at frame 0 needs frames $-2$ and $-1$, which do not exist. A causal filter invents them in the past only, for example as copies of frame 0. Then output 0 is a weighted sum of three copies of frame 0, and output 1 mixes frames 0, 0, and 1. No output ever reads a frame later than its own. A centered filter instead reads one frame back and one frame ahead, so output 0 already depends on frame 1.

To write this down, take one pixel and list its features over time as $\mathbf{x}_0, \dots, \mathbf{x}_{F-1} \in \R^{C}$, one vector of $C$ channels per frame. Spatial filters act the same way at every frame, so they change nothing below.

In plain words, output $j$ is a learned mix of the frame at position $js$ and the $k - 1$ frames just before it. The stride $s$ is how far the window jumps between outputs, so a stride of 2 halves the number of frames.

In plain words, a causal encoder reads the clip the way a viewer watches it, never ahead. Its first latent frame is a pure image latent. Each later latent frame summarizes the next $f_t$ frames, given everything before them. Asked about a video autoencoder in an interview, state this shape rule first, $1 + F_0$ frames in and $1 + F_0/f_t$ latent frames out.

Both versions give 3 latent frames, $1 + 8/4$. Each row of a table is one latent frame, and a 1 marks an input frame that changes it. In the causal table, row $j$ stops at frame $4j$, as the theorem says, and latent frame 0 sees only frame 0. In the centered table, latent frame 0 already depends on frames 0 to 3.

A full video autoencoder puts such convolutions, together with spatial ones, inside a variational autoencoder. A variational autoencoder (VAE, §32.3) is an autoencoder whose encoder outputs a mean and a variance for each latent number. It is trained with a small extra penalty, a KL divergence, that keeps the latents close to a standard Gaussian, so a generator has a well-shaped space to work in.

The causal design buys three things.

• Images and videos share one latent space. So one denoiser can train on both. That matters, because image-text data is far larger and more varied than video-text data. MAGVIT-v2 (Yu et al. 2023) built its video tokenizer from causal 3D convolutions for this reason. The Wan report (Wan Team 2025) says its first frame is "only spatially compressed", following MAGVIT-v2.
• Encoding and decoding can run chunk by chunk. Wan's autoencoder, Wan-VAE, splits a video of $1 + F_0$ frames into $1 + F_0/4$ chunks, one per latent frame. For each chunk it keeps the last two frames of features from the previous chunk in a cache, and the first chunk starts from two frames of zeros. Two frames is exactly the $k - 1 = 2$ past frames that a kernel of length 3 needs, so videos of any length fit in memory.
• Image-to-video comes almost for free. The first latent of a clip is the given image's own latent, which we use later.

The table lists the autoencoders of the systems in this section. The compression ratio is the number of pixel values per latent value, $3f_tf_s^2/c$.

The ratio column ignores the first frame. Because that frame is compressed in space only, the exact ratio is $\frac{3f_s^2}{c}\cdot\frac{1 + F_0}{1 + F_0/f_t}$. For Wan's 81 frames that is $12\cdot 81/21 = 46.3$ rather than 48.

The same two lessons as for images (§34.1) hold here. Temporal compression lets the same latent budget hold 4 or 8 times more frames, and more channels buy back reconstruction quality. CogVideoX measured both (Yang et al. 2024, Table 1).

import numpy as np

def causal_temporal_conv(x, kernel, stride=1, pad="replicate"):
    k = kernel.shape[0]
    l = (k - 1) // 2                               # placeholder: centered padding, which peeks ahead
    xp = np.concatenate([np.repeat(x[:1], l, 0), x, np.repeat(x[-1:], k - 1 - l, 0)], 0)
    n_out = (xp.shape[0] - k) // stride + 1
    return np.stack([np.einsum("khwc,kcd->hwd", xp[j * stride:j * stride + k], kernel) for j in range(n_out)])

The last test shows what replicate padding buys. Causality alone makes the first output of any clip equal to the encoding of its first frame on its own, under either padding. Replicate padding adds that every output of a still video equals that same image encoding, because the padded frames are copies of the real one. Zero padding makes the first $k - 1$ outputs differ from the rest, since their windows hold zeros where later windows hold the picture. So with replicate padding an image is exactly a video that never moves, and the image data and the video data agree about what a frame looks like.

Space-time patches and the token count

The denoiser is a transformer, and a transformer needs tokens. It gets them by cutting the latent video into small bricks, a few latent frames deep and a few positions on a side, and turning each brick into one vector. These bricks are the generative counterpart of the tubelets of §29.2, which showed that cutting and embedding bricks this way is a 3D convolution whose stride equals its kernel size. Sora's technical report (OpenAI 2024) described its own model only at this level. It compresses video into a latent space, cuts the latent into "spacetime patches" that act as the transformer's tokens, and treats images as videos with a single frame.

Nearly every documented video transformer uses $(p_t, p_s, p_s) = (1, 2, 2)$, and the Step-Video report calls $2\times2\times1$ latent patches the typical choice. With a Wan-style autoencoder ($f_t = 4$, $f_s = 8$), one token then covers 4 frames of $16\times16$ pixels (one frame for the tokens of the first latent frame).

Each token also needs to know where and when it is. Rotary position embeddings (RoPE, §5.5) rotate pairs of a query's and a key's coordinates by angles proportional to position, so attention scores depend only on relative position. The 3D version (§29.2) gives each token a (time, row, column) triple and assigns some coordinate pairs to each of the three. HunyuanVideo's 128-dimensional heads give 16 dimensions to time and 56 to each spatial axis (Kong et al. 2024). CogVideoX-5B and Wan's released code also use 3D RoPE. Movie Gen instead adds learned position vectors, one table each for time, row, and column.

The cell repeats the count for several published configurations.

LTX-Video stands out. Its autoencoder already compresses $8\times32\times32$ with 128 channels, so it uses no further patching. Five seconds of $768\times512$ video at 24 fps (121 frames) is then only 6,144 tokens. That is how it generates such a clip with 20 denoising steps in about 2 seconds on one H100 GPU, faster than the clip plays (HaCohen et al. 2024).

The token view also explains a practical finding of Sora's report. Sora trains on videos at their native durations, resolutions, and aspect ratios. Against a version trained on square crops, keeping the native aspect ratio improved framing and composition, where the square-crop model often cut the subject off. At inference, the report says, "we can control the size of generated videos by arranging randomly-initialized patches in an appropriately-sized grid". In token terms, a clip of any shape is a list of tokens with 3D positions. Clips of different shapes can share a training batch by packing their tokens into one sequence and masking attention between them, the approach of NaViT (Dehghani et al. 2023) that §31.5 describes.

def video_latent_shape(F, H, W, f_t, f_s, c, causal=True):
    return (F // f_t, H // f_s, W // f_s, c)          # placeholder: forgets the separate first frame

def video_tokens(F, H, W, f_t, f_s, patch=(1, 2, 2), causal=True):
    raise NotImplementedError

When attention dominates the cost

A transformer layer does two kinds of work. The linear part multiplies every token by weight matrices, in the four attention projections and in the MLP, so its cost grows in proportion to the token count $T$. The attention core compares every token with every other token, so its cost grows with $T^2$. For short sequences the linear part dominates, and for long ones attention does. The question is where the crossover lies.

Here is the count for one block, at $T = 1{,}000$ tokens of width $d = 64$ with an MLP of hidden width $4d = 256$. We count floating-point operations (FLOPs), two per multiply-add. Each of the four $d\times d$ projections costs $2Td^2$, and the two MLP matrices cost $2T\cdot d\cdot 4d$ each, so the linear part is $24Td^2 = 98$ million FLOPs. The attention core forms the $T\times T$ scores $\mathbf{Q}\mathbf{K}^\top$ and then the weighted sum of values, each $2T^2d$, so $4T^2d = 256$ million FLOPs. Attention already costs 2.6 times the rest, because $1{,}000$ is more than $6d = 384$. The same count appears for images in §34.1 and for video encoders in §29.3. Here we state it for any MLP width, including the gated MLPs (SwiGLU, §5.3) that some video transformers use, which have three weight matrices instead of two.

In plain words, once a clip has more tokens than about six times the model width, more than half of every layer's compute goes to tokens comparing themselves with other tokens. The share keeps climbing toward 1 as clips get longer. For video models this is the normal operating point, which is why nearly every efficiency idea for video targets attention. An interviewer who asks where the compute goes in a video model wants this crossover and one real number.

The cell computes the crossover, the forward FLOPs of the whole denoiser for one step, and the memory a stored score matrix would need, for four published models. It counts only the self-attention blocks over video tokens. Movie Gen and Wan also cross-attend to the text, which adds about $4Td^2$ per layer, and HunyuanVideo carries the text tokens inside the same sequence.

One denoising step of Movie Gen is about 9.6 PFLOP ($9.6\times10^{15}$ FLOPs). The score matrix of a single head at full length, stored in bf16 (2 bytes a number), would take 10.9 GB, and there are 48 heads in each of 48 layers. No production model stores it. FlashAttention (§12.3) computes exact attention tile by tile without ever storing the scores. Context parallelism (§10.6, §12.7) splits the sequence across GPUs. One form, ring attention, passes blocks of keys and values around a ring of GPUs. Another, Ulysses, swaps the split so each GPU holds whole sequences for a subset of heads. Movie Gen trained with tensor, sequence, context, and fully sharded data parallelism together, and Wan combines Ulysses and ring attention. These methods remove the memory wall, but they do not remove a single FLOP.

The widget runs the whole chain for a clip you choose. Each model opens with the clip it was built for, at its own frame rate and with its own autoencoder. Then change the duration, the resolution, and how many video frames the autoencoder packs into each latent frame. The widget shows the latent, the token count, and where attention's share of the compute passes one half.

def layer_flops(T, d, d_ff, gated, T_c=0):
    linear = T * (4 * d * d + (3 if gated else 2) * d * d_ff)   # placeholder: counts multiply-adds
    attention = 2 * T * T * d
    return linear, attention

def crossover(d, d_ff, gated):
    raise NotImplementedError

One clip from prompt to frames

We now have every piece, so let us follow one generation from a prompt to frames. Take Wan 2.1 14B making five seconds of 480p video, 81 frames of $832\times480$, from a one-sentence prompt such as "a red fox trots across fresh snow at dawn". Every shape below follows from the configuration counted above.

1. Text. The prompt is split into tokens and read by a frozen text encoder, umT5, a multilingual version of the T5 encoder (Wan Team 2025). It returns up to 512 token vectors of 4,096 numbers each, which a small MLP maps to the model width. How these vectors steer the picture is the subject of §34.3.
2. Noise. The sampler draws a latent video of pure Gaussian noise, $21\times60\times104\times16$, the shape the autoencoder would give for 81 frames. Time is already compressed, so each of the 21 latent frames will become 4 video frames, except the first, which becomes one.
3. Tokens. Patches of $(1, 2, 2)$ cut the latent into 32,760 bricks of $1\cdot2\cdot2\cdot16 = 64$ numbers. One learned matrix maps each brick to a token of width 5,120, and each token gets its (time, row, column) position through 3D RoPE.
4. Denoising. The 40 blocks run full self-attention over all 32,760 tokens and cross-attention from them to the text vectors. The current noise level enters every block through adaptive normalization, a shift, scale, and gate computed from the noise level (§34.2). The output is unpatchified back to a $21\times60\times104\times16$ prediction of the velocity, the difference between noise and clean latent along the straight flow-matching path (§33.6).
5. Guidance and steps. Classifier-free guidance (§33.5) runs the network twice per step, once with the prompt and once with an empty prompt, and pushes the prediction away from the unconditional one. The sampler takes a small step toward clean data and repeats. Take $S = 50$ steps, the setting CogVideoX and Movie Gen report.
6. Decoding. The clean latent goes through the Wan-VAE decoder chunk by chunk, latent frame 0 to video frame 0 and each later latent frame to the next 4 frames. The result is $81\times480\times832\times3$ pixel values, the fox's five seconds.

The widget steps through the same pipeline, stage by stage and then denoising step by denoising step. It shows Wan's real shapes beside a toy clip of 9 frames that is compressed and denoised the same way. Press Play to run it, or drag the stage slider to stop at any stage.

The cell traces the shapes and prices the denoiser, with the text cross-attention included this time.

Two things stand out in the trace. The latent the model works on has 2.1 million numbers, 46 times fewer than the 97 million pixel values it decodes to. And almost all the time goes to step 4, repeated 100 times. That is why the rest of this section is about which tokens attend to which, and about taking fewer steps.

Which tokens each token sees

The quadratic cost invites limiting which tokens each token may attend to. The options form a short list, and the history of video generation is a migration along it.

Temporal layers inside an image model. The first diffusion video models kept a 2D image network and inserted layers that mix across time. Video Diffusion Models (Ho et al. 2022a) used a 3D U-Net factorized over space and time. Spatial attention ran inside each frame, then temporal attention ran across frames at each position. It trained on videos and images together by appending independent images to each video and switching off temporal attention for them. Make-A-Video (Singer et al. 2022) added space-time layers to a text-to-image model and learned motion from unlabeled video, with no paired video-text data at all. Video LDM (Blattmann et al. 2023a) turned Stable Diffusion into a video model by training only inserted temporal layers. One appeal is reuse, since the expensive image model is already trained. The other is cost. Write $n_s = hw/p_s^2$ for the tokens per latent frame. Factorized attention scores $T(n_s + F')$ pairs instead of $T^2$.

Full 3D attention. Every token attends to every other token of the clip. CogVideoX compared it with "2D + 1D" factorized attention at the same model size. The factorized model reached much worse FVD, a video quality score defined in §34.6, early in training, and was "unstable and prone to collapse" (Yang et al. 2024). The reason is reachability, the argument of §29.3. In a factorized block, temporal attention connects only tokens at the same spatial position. So a patch of a moving object in frame $i+1$ cannot attend directly to where the object was in frame $i$. The information has to hop through intermediate tokens, and large motion needs many hops. HunyuanVideo chose full attention for the same reason, and because it treats images and videos alike and reuses the acceleration tools built for long-context language models (Kong et al. 2024).

The price is real but smaller than the pair count suggests. One forward step of CogVideoX on an H800 at $480\times720$ and 6 seconds took 2.11 seconds with full attention and 1.26 seconds factorized, and 9.60 against 4.17 seconds at $768\times1360$ (Table 8 of the same paper). Since 2024 the documented frontier models use full 3D attention and pay that price.

Local and causal patterns. Between the two extremes sits windowed or sparse attention, where each token sees a space-time neighborhood (the 3D version of the local attention of §12.6). And there is the pattern that streaming generation needs, causal attention in time.

Here is that last pattern on a small case. Take 6 latent frames and group them into chunks of 2, so frames 0 and 1 form chunk 0, frames 2 and 3 form chunk 1, and so on. A token in chunk 1 may see every token of chunks 0 and 1, including the other frame of its own chunk. It may not see chunk 2. That is block-causal attention.

The point of this mask is that a model built on it never looks at the future. So the future can be generated later, chunk by chunk, with a cache, and the outputs are exactly the same as one masked pass over the whole clip. The cache is the KV cache of language models (§6.5), which stores the keys and values of every token already processed so that each new token computes only its own.

Proof. Induct on the layers. Suppose the inputs to layer $\ell$ agree in both procedures for every token in chunks $0, \dots, a$. This holds for $\ell = 1$, where the inputs are the latent tokens themselves. Under the mask, a query in chunk $a$ sees exactly the keys and values of chunks $0, \dots, a$, computed from those identical inputs. In the streaming procedure, when chunk $a$ is processed the cache holds exactly the keys and values of chunks $0, \dots, a$, from the same inputs. The masked-out scores contribute $e^{-\infty} = 0$ to the softmax, so both softmaxes run over the same scores and give the same weights and outputs. The MLP and the normalizations act on each token separately, so the inputs to layer $\ell + 1$ agree for chunks $0, \dots, a$. ∎

In plain words, this is the language-model KV cache with chunks of frames in place of single tokens. In a diffusion model the theorem applies to each network call. A chunk is denoised in several calls against the same cache of finished chunks, and only then are its own clean keys and values added. The cell checks the theorem numerically.

With 3 chunks of 8 tokens, the mask keeps $8\cdot8 + 8\cdot16 + 8\cdot24 = 384$ of 576 pairs. In general, $C$ chunks of $m$ tokens keep $m^2C(C+1)/2$ pairs, a fraction $(C+1)/2C$ of full attention. So causality alone saves at most half the work. Its value is latency, the time until the first frames appear, not FLOPs.

The widget shows each pattern on a small clip, 6 latent frames of 4 tokens as in the cell. Each small square is one latent frame, and the matrix beside it is the whole mask, one row per query and one column per key. In factorized mode it colors the keys of the spatial layer and of the temporal layer separately. Its key count adds the two layers, so the query's own token counts once in each, as in the pair count $T(n_s + F')$.

The table collects the patterns. Here $n_s$ is the number of tokens per latent frame and $F'$ the number of latent frames.

import numpy as np

def block_causal_mask(F_lat, tokens_per_frame, chunk):
    T = F_lat * tokens_per_frame
    return np.tril(np.ones((T, T), dtype=bool))      # placeholder: token-causal, which is too strict

def streaming_attention(Q, K, V, F_lat, tokens_per_frame, chunk):
    raise NotImplementedError

The recipe the large systems share

Before 2024 the best video models were chains of several networks. Since 2024 they are one latent transformer. The history box traces that change. The rest of this part reads the recipe from the systems that published it.

OpenAI's technical report on Sora, of February 15, 2024, describes the recipe in words and says that "model and implementation details are not included" (OpenAI 2024). Here is what it does state.

• A video compression network maps video to a latent compressed in time and space.
• The latent is cut into spacetime patches that serve as transformer tokens, so images are one-frame videos.
• Sora is a diffusion transformer trained to predict clean patches from noisy ones, and sample quality improved markedly with training compute.
• It trains on data at native durations, resolutions, and aspect ratios. It generates up to a minute of high-definition video, and images up to $2048\times2048$.
• It uses the recaptioning technique of DALL-E 3 (§34.3), training on long captions written by a video captioning model, and expands short user prompts into detailed captions with GPT.
• It can be prompted with images and videos. It can animate an image, extend a video forward or backward in time, edit a video by partly noising and re-denoising it (SDEdit, §34.7), or interpolate between two videos.

Every piece is in this chapter. The open systems below fill in the numbers.

All of them train with flow matching or a close relative (§33.6), except CogVideoX, which uses v-prediction. Three training habits recur.

Images first, then video, low resolution first. Movie Gen pretrains on text-to-image at 256 px. It then trains jointly on images and videos at 256 px and then at 768 px, with one image sample for every ten video samples. Wan pretrains its 14B model on 256 px images. It then trains jointly in three stages, 256 px images with 192 px videos, then 480 px, then 720 px. Starting from images teaches appearance and text alignment on short sequences, so the expensive long sequences only have to teach motion. Movie Gen reports that training jointly from scratch converged much more slowly than starting from a text-to-image model.

Curate for motion. Stable Video Diffusion's raw collection had 580M clips, 212 years of video, and was filtered to 152M clips. The filters removed near-static clips by their mean optical-flow magnitude (computed at 2 fps), clips with too much text, and clips with low aesthetic or text-image scores. Movie Gen keeps 4 to 16 second single-shot clips with "non-trivial motion". It removes static videos with a detector, keeps clips with reasonable motion scores, removes clips with frequent jittery camera moves, and drops special effects such as slideshows. A model trained on what the internet uploads learns that most videos barely move.

Spend the compute on the last stage, and save it at inference. Most of Movie Gen's training FLOPs go to the 768 px stage at 73K tokens, even though that stage runs fewer iterations than the 256 px stage. Each of its samples has 9 times the tokens of a 256 px sample, and attention grows with the square of that count (Polyak et al. 2024). At inference, cost is cut mainly by taking fewer steps. Movie Gen's linear-quadratic schedule emulates an even schedule of $N$ steps with only 50. Its first 25 steps sit exactly where the first 25 steps of the $N$-step schedule would, at high noise where the output changes most, and 25 steps spaced quadratically cover the rest of the way. In practice the authors use $N = 250$, so 50 steps stand in for 250. HunyuanVideo shifts its timesteps toward high noise (the shift of §34.1), and shifts them further the fewer steps it takes.

Conditioning on an image and extending a clip

Image-to-video means animating a given picture. The model must keep the picture as the first frame and invent the motion after it. The usual way is to feed the image's latent to the denoiser as extra input channels, next to the noisy latent of the clip.

Stable Video Diffusion does this in two places. It replaces the text embedding in cross-attention with the CLIP image embedding of the conditioning frame. And it concatenates a noise-augmented copy of the frame's latent, channel by channel, to the noisy input of every frame. It found that a constant guidance scale was either too weak (frames drift from the given image) or too strong (colors oversaturate). So it raises the guidance scale linearly across frames instead.

Wan's image-to-video model is more general. It concatenates three things along the channel axis.

• The noisy latent, $c = 16$ channels.
• The condition latent, another $c = 16$ channels. It is the Wan-VAE encoding of a clip whose first frame is the conditioning image and whose other frames are zeros. Because the autoencoder is causal, its first latent frame is the image's own latent.
• A binary mask marking which frames are given, rearranged so each latent frame carries the mask values of its $f_t = 4$ video frames as 4 channels.

So its first layer reads $2c + f_t = 36$ channels instead of 16. The new input channels enter through a projection initialized to zero, so the image-to-video model starts out as the text-to-video model. CLIP image features enter through a separate cross-attention. The same mask lets one model do image-to-video, continuation, and interpolation between a first and a last frame, depending on which frames are marked as given (Wan Team 2025).

Noise augmentation of the condition is the same trick that cascades use for their low-resolution inputs (§34.1). At inference the conditioning frame may itself be generated, with artifacts a real frame never has. A little noise during training teaches the model not to copy those details blindly.

Extension without retraining is harder. Suppose a model was trained only on unconditional clips, and we want it to continue a given clip $\mathbf{x}^a$ with new frames $\mathbf{x}^b$. The tempting replacement method samples the whole clip and, at every step, overwrites the known frames with a freshly noised copy of $\mathbf{x}^a$ at the current noise level. Video Diffusion Models (Ho et al. 2022a) found that it gives continuations that look fine alone but do not follow the given frames.

They also explained why. Write $\mathbf{z}_t$ for the noisy clip at noise level $t$. The update of the new frames moves toward the model's estimate $\mathbb{E}[\mathbf{x}^b\mid\mathbf{z}_t]$. But a correct conditional sampler needs $\mathbb{E}[\mathbf{x}^b\mid\mathbf{z}_t, \mathbf{x}^a]$, which uses the clean given frames, not only their noisy copy. Their fix, reconstruction guidance, adds the missing term approximately. It nudges the new frames along the gradient of how well the model's own reconstruction of the known frames matches them.

Write $\hat{\mathbf{x}}^a(\mathbf{z}_t)$ and $\hat{\mathbf{x}}^b(\mathbf{z}_t)$ for the model's estimates of the clean known and new frames, and $\alpha_t$ for the weight of the clean data in $\mathbf{z}_t$. The adjusted estimate of the new frames is

where $\mathbf{z}_t^b$ is the noisy part of the clip that holds the new frames and $w_r$ is a weight. In plain words, the new frames move in whatever direction makes the model reconstruct the given frames better. The weight $w_r = 1$ is what a Gaussian approximation of $q(\mathbf{x}^a\mid\mathbf{z}_t)$, the distribution of the given frames given the noisy clip, produces.

Even with a perfect denoiser, replacement continues the clip with mean 0.89 instead of 1.80 and variance 0.43 instead of 0.19. It captures only about half of what the given frame implies about the next one. Reconstruction guidance with $w_r = 1$, the Gaussian weight, moves the mean to 1.17. Larger weights move it further, to 1.75 at $w_r = 4$, while shrinking the variance below the right value. That matches what Ho et al. report for real videos, that $w_r > 1$ gives better samples. It also shows that this guidance weight, like others, trades diversity for fidelity.

What this changes in practice is that production systems avoid the problem by training with masks, as Wan does. The model then learns $\mathbb{E}[\mathbf{x}^b\mid\mathbf{z}_t, \mathbf{x}^a]$ directly. An interviewer who asks how to extend a clip with an unconditional model wants the replacement method, why it fails, and the masked-training fix.

Keeping a clip consistent

Most visible failures of video models have a specific cause, and most have a known fix.

• Flicker from the decoder. Decoding frames one by one, or a long clip in separate temporal tiles, puts seams in time. Movie Gen decodes in tiles of 32 frames (4 latent frames) that overlap by 16 frames, and blends neighboring tiles linearly. HunyuanVideo found that tiling at inference without training for it left visible artifacts, so it fine-tuned its VAE with tiling switched on at random.
• Weak motion. Static training data gives static samples. That is why curation filters by motion. It is also why Stable Video Diffusion conditions on the frame rate and a motion score, so motion can be turned up at sampling time. Movie Gen conditions on frame rate by prepending it to the prompt, as in "FPS-16".
• Identity drift. Over long clips, and especially over repeated extensions, faces and objects change slowly. Full attention over the whole clip, reference images in context, and training on the model's own rollouts (below) all help, and none is a guarantee.
• Physics and object permanence. Sora's report lists its own limitations. It "does not accurately model the physics of many basic interactions, like glass shattering", eating food does not always change the object's state, and long samples develop incoherencies and objects that appear from nowhere. OpenAI's Sora 2 announcement (OpenAI 2025, September 30, 2025) describes earlier models as "overoptimistic", bending reality to satisfy a prompt, for example a missed basketball shot teleporting into the hoop.

How these failures are measured, with FVD and VBench, is the subject of §34.6.

Generating in real time

A full-attention video model cannot show the first frame until it has finished the last one, because every step updates all frames together. CausVid's bidirectional teacher took 219 seconds to generate a 10-second clip at $640\times352$ (Yin et al. 2024). Interactive uses, such as live generation, games, and robotics, need frames to come out one after another, the way a language model streams tokens. Block-causal attention makes that possible. The question is how to train such a model so its frames stay good.

The obvious recipe is teacher forcing. Train a block-causal model to denoise each chunk given the clean ground-truth chunks before it, then generate one chunk after another. It inherits exposure bias (§3.5). In training the past is always real. At inference the past is the model's own imperfect output, a kind of input it never trained on, so errors compound over the rollout. Autoregressive image generation faces the same problem token by token (§34.4).

The first fix lets every frame carry its own noise level during training.

One model trained this way covers a whole family of samplers. If all frames share one noise level, it is a full-sequence diffusion model. If each frame is denoised completely before the next starts, it is next-frame prediction. In between, a lagged (or pyramid) schedule starts frame $j$ a fixed number of network calls $\Delta$ after frame $j-1$. Near frames are then nearly clean while far frames are still noise.

Here is a small case. With $S = 4$ steps per frame and a lag of 2, frame 0 is denoised during calls 1 to 4, frame 1 during calls 3 to 6, and frame 2 during calls 5 to 8. Frame 0 is ready after 4 calls instead of waiting for the whole clip.

Diffusion Forcing also steadies long rollouts by conditioning on a slightly noised version of the generated past rather than the past itself, which hides small errors from the model. And its authors prove that the training objective is a lower bound on the likelihood of every subsequence of frames.

import numpy as np

def pyramid_schedule(F_lat, S, lag):
    M = S + lag * F_lat                               # placeholder: one call too many per frame of lag
    m = np.arange(M + 1)[:, None]
    j = np.arange(F_lat)[None, :]
    return 1.0 - (m - lag * j) / S                    # and no clipping

The exercise exposes the trade that a schedule controls. With lag 0 the first frame waits for all $S$ calls, like every other frame. With lag $\Delta$ the first frame is ready after $S$ calls, and each later frame $\Delta$ calls after the one before. But the total number of calls grows to $S + \Delta(F' - 1)$. Streaming lowers latency and costs more calls in total, unless each call is cheap. Few-step students and caches make calls cheap.

CausVid (Yin et al. 2024) makes the causal model fast as well as causal. Its student starts as the pretrained bidirectional transformer with a block-causal mask. It is first fitted by regression to pairs of noise and clean output taken from the teacher's sampling ODE. Then it is trained with distribution matching distillation (DMD, §33.4), which pushes the distribution of the student's outputs toward the teacher's, using the difference of two score estimates. This turns the 50-step bidirectional teacher into a 4-step causal generator. During training each chunk gets an independent noise level, as in Diffusion Forcing. A bidirectional teacher can supervise a causal student because DMD matches only the distribution of whole clips, not each step.

At inference each chunk is denoised in 4 steps against the cache. Then one more pass over the clean chunk, at noise level 0, writes its keys and values to the cache for later chunks. CausVid streams at about 9.4 frames per second after an initial latency of 1.3 seconds on one GPU.

Self Forcing (Huang et al. 2025) attacks exposure bias directly. During training the student rolls out a whole video chunk by chunk with its KV cache, exactly as at inference. A distribution-matching loss (DMD, a related score-based distillation loss, or a GAN loss) is then applied to the finished video, so the student learns from its own past. To keep this affordable it denoises in 4 steps and lets gradients flow through only one randomly chosen step. A rolling KV cache, which keeps a fixed window of recent frames, allows videos longer than the training length.

Self Forcing builds on Wan 2.1 1.3B at $832\times480$. Its chunk-wise model, with 3 latent frames per chunk, streams at 17.0 frames per second with 0.69 seconds of latency on one H100. The base model runs at 0.78 frames per second with 103 seconds of latency. A frame-wise version reaches 0.45 seconds of latency at 8.9 frames per second.

The same machinery, conditioned on a player's or a robot's actions as well as on text, turns a video generator into an interactive simulator that predicts the next frames given what the agent does. Such action-conditioned video world models reuse the block-causal attention and streaming caches of this section.

The compute bill and the frontier

Training cost follows from the per-step count. A training step costs about three forward passes, one forward and a backward pass that costs about twice as much (§0.2). So one 16-second Movie Gen clip at 768 px costs about $3\times9.6 = 29$ PFLOP to train on. At a sustained 400 TFLOP/s per GPU, that is 72 GPU-seconds per training clip, and Movie Gen trained on up to 6,144 H100s. Sampling one clip with 50 steps costs $50\times9.6 = 480$ PFLOP, about 20 GPU-minutes at the same rate, before the spatial upsampler that takes it to 1080p. Guidance run as a second pass doubles it. These are estimates from the formula, not measurements.

The frontier now generates sound with the picture. Google's Veo 3, announced May 20, 2025, generates "videos with audio", from background noise to dialogue between characters (Google 2025). OpenAI's Sora 2, announced September 30, 2025, is described as a "video and audio generation model" that creates background soundscapes, speech, and sound effects (OpenAI 2025). Neither company has published its architecture.

The documented route to sound is Movie Gen Audio, a separate 13B model that generates audio conditioned on the video and the text. How audio becomes tokens is the subject of §37.3, on neural audio codecs, and how a model generates those tokens is the subject of §37.4, on codec language models. There the latent-plus-generator pattern of this chapter returns for sound.

In PyTorch

The reference block builds a small causal 3D encoder with F.pad, which pads time in the past only by replicating the first frame. It checks the shape rule and causality with autograd. Then it checks that streaming chunk by chunk with F.scaled_dot_product_attention equals one masked pass.

Latent frame 1 is reached by input frames 0 to 4 and no later, $jf_t = 4$ for $j = 1$, and 17 frames give 5 latent frames. Production code adds a feature cache, so each chunk reuses the last $k - 1$ frames of features from the previous one instead of encoding them again.

Read more

• Ho et al. (2022). Video Diffusion Models. The factorized space-time U-Net, joint image and video training, and reconstruction guidance for extension, with the analysis of why replacement fails.
• Blattmann et al. (2023). Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. The three training stages, motion-aware data curation, and image-to-video conditioning in detail.
• OpenAI (2024). Video generation models as world simulators. Sora's technical report, the recipe in words and the model's own list of limitations.
• Yang et al. (2024). CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. A causal 3D VAE, separate adaptive normalization for text and video, the ablation of full against factorized attention, and measured inference times.
• Polyak et al. (2024). Movie Gen: A Cast of Media Foundation Models. The most complete public account of a frontier video model, from the autoencoder and tiling to parallelism, data curation, and the audio model.
• Wan Team (2025). Wan: Open and Advanced Large-Scale Video Generative Models. An open 14B model with its causal VAE and feature cache, cost model, parallelism, and unified image-to-video masking.
• Chen et al. (2024). Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. Independent per-token noise levels and the schedules they allow.
• Huang et al. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. Training a causal few-step student on its own rollouts for real-time streaming, building on CausVid (Yin et al. 2024, From Slow Bidirectional to Fast Autoregressive Video Diffusion Models).

Watch next

▶

▶