This section is about masked video modeling, a way to train a video network without labels. You hide most of a clip, show the network the rest, and train it to fill the hidden part back in. Labeled video is scarce and expensive, while unlabeled video is nearly free. So a training signal that comes from the clip itself is worth a great deal. The recipe this section builds is VideoMAE, the masked autoencoder for video (Tong et al. 2022).

Here is how much it matters. Something-Something v2 is a dataset of 169,000 labeled training clips of people doing 174 actions with everyday objects, such as "pushing something from left to right". The label depends on the motion, not on which objects are in the frame. The network is ViT-B, the base-size Vision Transformer, which cuts its input into small pieces, turns each piece into a vector called a token, and runs 12 Transformer layers of width 768 over the tokens (§27.2). Trained from scratch on those clips, it gets 32.6 percent top-1 accuracy, the share of validation clips whose highest-scoring class is the right one.

Now pretrain the same network on the same clips without their labels, by hiding 90 percent of each clip and training it to fill the hole. Then fine-tune it, which means training all its weights further on the labeled clips. It reaches 69.6 percent. No data was added. Only the training signal changed.

Masking also makes a costly network affordable. §29.3 ended on the default video encoder of 2026, a plain Vision Transformer whose attention connects every token of a clip with every other token, across space and time. On the 1,568 tokens of a standard clip that is expensive. During masked pretraining the encoder reads only the tenth of the clip that stays visible, so it costs about a tenth as much. Fine-tuning and inference still see every token.

The method comes from images. The masked autoencoder of §28.5 (He et al. 2022) cuts an image into square patches and hides most of them. An encoder reads only the visible patches, and a small decoder rebuilds the pixels of the hidden ones. Images work best with 75 percent of the patches hidden. The idea itself comes from text, where the encoder BERT hides 15 percent of the words of a sentence and predicts them from the rest (§7.1).

Video adds one trap. Consecutive frames are nearly the same picture, as §29.1 measured. So a hidden piece of a clip usually has a near copy at the same place a fraction of a second earlier or later. Hide 75 percent of a 16-frame clip at random, and 87 percent of the hidden pieces still have a visible piece at the same place at some other time. A network can then lower its error by copying from the same place at another time. That teaches it almost nothing about what is in the video.

This section makes the trap exact and shows the two ways out. One is to hide far more of the clip, 90 percent. The other is to hide the same places in every frame, which is called tube masking. Then it writes the loss, derives what masking saves in compute, and checks the derivation against published counts to within one percent. It goes through VideoMAE's results and its claim to data efficiency, the decoder masking that scaled it to a billion parameters, and what predicting pixels cannot teach. It ends with masks that move with the content, which stop copies along motion, and with targets other than pixels.

Masked autoencoding for a clip

A video network reads a clip as tokens, so start there. §29.2 cut a clip into tubelets, small bricks a few frames long and a square of pixels wide, and turned each brick into one token with one learned matrix. A clip $\mathbf{X}\in\R^{T_f\times H\times W\times C}$ has $T_f$ frames of height $H$, width $W$, and $C$ color channels. A tubelet spans $P_t$ frames and $P\times P$ pixels.

The bricks form a grid. Along time there are $n_t = T_f/P_t$ time slices. Each slice has $n_h = H/P$ rows and $n_w = W/P$ columns, so $n_s = n_hn_w = HW/P^2$ places, the spatial positions of a slice. So there are $T = n_tn_s$ tokens.

VideoMAE's standard input is 16 color frames of $224\times224$ pixels with $2\times16\times16$ tubelets. That makes $n_t = 16/2 = 8$ time slices and $n_s = 14\cdot14 = 196$ places, so $T = 1{,}568$ tokens. We index a token by its time slice $i\in\{0,\dots,n_t-1\}$ and its place $j\in\{0,\dots,n_s-1\}$. We write $\mathbf{x}_{ij}\in\R^{D}$ for its raw pixels, where $D = P_tP^2C = 2\cdot16\cdot16\cdot3 = 1{,}536$ values.

In plain words, the encoder sees a tenth of the clip. The decoder is told where the other nine tenths were, through the mask tokens and their positions. The network is graded only on how well it fills them in. At VideoMAE's 90 percent, the encoder reads 160 tokens and the decoder reads all 1,568.

Two facts carry over from images unchanged. First, the encoder never processes a mask token, so its cost depends on the $T_v$ visible tokens and not on $T$. That is where the compute savings later in the section come from. Second, the encoder is judged after fine-tuning. Pixel reconstruction gives features that are weak when they are frozen, kept fixed and read by a simple classifier trained on top. §28.5 found this for images, and the last part of this section finds it again for video.

A hidden tubelet often has a visible twin

The first design choice is which tokens to hide. The published methods use three patterns.

The name comes from the picture. Stack the slices of a clip on top of each other, and each hidden place becomes a hidden tube running through time.

Call a token a twin of a hidden token if it sits at the same place $j$ in a different time slice. If the scene does not move, a twin has the same pixels, so copying it rebuilds the hidden token perfectly. So the question is how often a hidden token has a visible twin. Start with a grid small enough to check by eye.

For general sizes the answer is a counting argument. Fix one hidden token and a set of $k$ other tokens, for example its twins. How likely is it that all $k$ are hidden as well? If they are, nothing in the set can be copied.

In plain words, once $z$ is hidden, each further token is slightly less likely than $\rho$ to be hidden, because $z$ has used up one of the hidden slots. So all $k$ are hidden with probability at most $\rho^k$, and at least one is visible with probability at least $1-\rho^k$. For example, with $T = 8$ and $\lvert\Omega\rvert = 6$ as in the two-by-four grid, a hidden token's twin is also hidden with probability $q_1 = 5/7 = 0.71$, a little below $\rho = 0.75$.

Two choices of $K$ matter. If the content changes slowly, the useful twins are the temporal neighbors $(i-1, j)$ and $(i+1, j)$, the same place one slice earlier and one slice later. That is $k = 2$ for an interior slice and $k = 1$ for the first and last slices. If the content is static, every token of the same tube is a perfect copy, so $k = n_t - 1$.

In plain words, the leak fraction averages, over the slices, the chance that a hidden token has a visible neighbor at its place. For large $T$ each $q_k$ is close to $\rho^k$, so the leak fraction is about $\big(2(1-\rho) + (n_t-2)(1-\rho^2)\big)/n_t$. At VideoMAE's $n_t = 8$ and $\rho = 0.9$ that is $(2\cdot0.1 + 6\cdot0.19)/8 = 0.168$. So about one hidden token in six has a visible neighbor at the same place. If the scene were static, more than half of the hidden tokens would have a perfect copy somewhere in their tube, since $1 - q_7\ge 1-0.9^7 = 0.52$.

Here is the leak fraction in code, checked on the two-by-four example.

Now the theorem at real scale, on VideoMAE's $8\times196$ grid, against a simulation of random masks.

The simulation matches the formula to within 0.001. Read the table from the bottom up. Even at 95 percent hidden, one hidden token in twelve has a visible neighbor at its place, and three in ten have a copy somewhere in their tube. At 75 percent, the image default, the neighbor leak is 39 percent. A static scene would then hand the decoder a copy for 87 percent of the hidden tokens.

A higher ratio shrinks the leak but never removes it. Tube masking removes it by construction. This is the first thing an interviewer probes about VideoMAE. Why tube masking, and why 90 percent rather than the image default of 75? The answer is this table, plus the caveat in the next part that tubes stop only copies that stay in place.

import numpy as np

def tube_mask(n_t, n_s, ratio, rng):
    raise NotImplementedError

import numpy as np

def leak_fraction(mask):
    raise NotImplementedError

def random_leak_formula(n_t, n_s, ratio):
    raise NotImplementedError

The copying shortcut, measured

The theorem counts chances to copy. Whether copying actually works depends on the content, so we measure it on a synthetic clip like those of §29.1. It has 16 grayscale frames of $64\times64$ pixels, a textured background, and a textured square moving 2 pixels per frame. We cut it into $2\times8\times8$ tubelets, so $n_t = 8$ and $n_s = 64$.

The copy baseline is a predictor that understands nothing. It fills each hidden token with the nearest visible token at the same place in time. When the token's whole tube is hidden, it uses the mean of the visible tokens in the token's slice. A hidden token counts as solved by copying when the baseline's error on it is below 5 percent of a reference error. The reference is the error of predicting the mean of all tokens in its slice, visible and hidden. We try a static camera and a camera that pans, sliding the view sideways by 2 pixels per frame.

With a static camera, random masking at 90 percent lets the baseline solve about half of the hidden tokens (0.48). The theorem predicts 0.52 for fully static content. The gap is the moving square, which cannot be copied in place. Frame masking at 87.5 percent keeps one time slice of eight. On the background it is even easier, since the visible slice holds a copy of every static place (0.85). Tube masking leaves nothing to copy (0.00). That is the whole argument for tube masking, and it is VideoMAE's argument.

The panning camera changes the picture. Same-place copying now solves nothing under any mask, because the content moved. That does not mean the shortcut is gone. The hidden content has moved to another place, where it may well be visible. Attention can learn to look up a displaced location as easily as the same one. VideoMAE says so itself, writing that tube masking relieves leakage "for cubes with no or negligible motion".

So tube masking fixes the static part of video, the backgrounds and the objects that barely move within the 2 seconds a clip covers. It does not make copying impossible. Exercise 29.4e near the end of the section measures this, and the masks that move with the content, right after it, close the gap. In practice the size of the leak depends on your footage. Static cameras leak the most under random masking, and handheld or panning cameras leak along the motion even under tubes.

The widget below runs the copy baseline on a smaller clip of 8 frames, one per time slice. With a panning camera it also runs a second baseline that knows the pan. It copies each hidden patch from the place its content occupied in another frame.

How much to hide

Which ratio is best is an empirical question, and the answer tracks how redundant the data are. Two groups answered it for video at the same time. One is VideoMAE. The other is a study from Meta titled Masked Autoencoders As Spatiotemporal Learners (Feichtenhofer et al. 2022), which we call MAE-ST. It trains the same kind of model with a larger encoder, ViT-L, which has 24 layers of width 1,024.

Kinetics-400, the second benchmark here, has about 240,000 training clips of 400 human actions from YouTube, such as playing a violin or riding a horse (§29.1). Most of its labels can be read off a single frame, from the scene or the objects. That makes it the appearance-heavy counterpart to the motion-heavy Something-Something v2.

The two video papers agree on the ratio and disagree about the pattern. An ablation changes one design choice and keeps everything else fixed. Their masking ablations are side by side below, as fine-tuned top-1 accuracy after 800 epochs of pretraining (an epoch is one pass over the training clips).

VideoMAE finds tube masking better than random masking, by 1.3 points on Something-Something v2 (SSv2) and 0.5 on Kinetics-400 (K400). MAE-ST finds random masking better than tube masking, by 0.9 points on Kinetics-400. Both find frame masking worst by far, and both find 90 percent best.

Why would random masking ever win? Because the leak is not the only thing a mask changes. Each pattern also decides how much of the scene the encoder sees at all. Tube masking at 90 percent never shows the encoder 90 percent of the places. Random masking shows a place in at least one slice unless all $n_t$ of its tokens are hidden. That happens with probability $\prod_{r=0}^{n_t-1}(\lvert\Omega\rvert - r)/(T-r)\approx\rho^{n_t} = 0.9^8 = 0.43$. So random masking covers about 57 percent of the places. Frame masking shows every place, but only at one or two times.

In plain words, every mask trades leakage against coverage. Random masking at 90 percent leaks to one hidden token in six but sees more of the scene. Tube masking leaks nothing in place but sees a tenth of it. Frame masking sees the whole scene at one moment and asks the model to predict seven others. MAE-ST calls that an overly difficult task of predicting past and future from a single frame.

Which of the first two wins depended on model size, data, and recipe in these two papers. Three lessons are robust. The ratio should be about 90 percent. Frame masking fails. And tube masking helped more on the motion-heavy Something-Something v2 than on the appearance-heavy Kinetics, the contrast §29.3 drew. So in an interview, the safe answer to "tube or random?" is to name the trade-off between leakage and coverage and to say which benchmark you care about.

The reconstruction target and the loss

The second design choice is what the decoder predicts. VideoMAE predicts the pixels of the hidden tubelets of the input clip itself. Each tubelet's pixels are first standardized on their own, shifted to mean 0 and scaled to standard deviation 1.

In plain words, each hidden tubelet is graded on its pattern, where it is brighter or darker than its own average, measured in units of its own contrast. It is not graded on its average brightness or its contrast. A tiny case shows the effect. A grayscale tubelet with pixels 0.2, 0.4, 0.6 has mean 0.4 and unbiased standard deviation 0.2, so its target is $-1, 0, 1$. The tubelet 0.5, 0.6, 0.7 has the same target, although it is brighter and flatter.

Why throw away each tubelet's brightness and contrast? Because they are the easy part. The average of a hidden tubelet is close to the average of its visible neighbors. So a decoder can get much of a raw-pixel loss right by blurring, and the gradient then mostly teaches blurring. Here is how much of the pixels' spread in our synthetic clip is only the spread of the per-tubelet means.

Of the total squared deviation of the pixels from the clip's mean, 38 percent is the deviation of the tubelet means alone. Interpolation from visible neighbors recovers that part cheaply, and the share grows as the content gets smoother. After normalization every target has mean 0 and standard deviation 1. The pooled value prints as 0.996 for a reason worth knowing. Each tubelet's unbiased estimate divides by 127, one less than the 128 values in this clip's $2\times8\times8$ grayscale tubelets, and $\sqrt{127/128} = 0.996$. So the whole loss is about structure inside the tubelet.

The ablations agree. MAE-ST gains 0.6 points on Kinetics-400 from normalized pixels, 84.4 against 83.8 for raw pixels, and VideoMAE uses them by default. VideoMAE also compares losses on Something-Something v2. Mean squared error gets 69.6. The L1 loss, the mean absolute error, gets 69.1. Smooth L1, which is squared for small errors and absolute for large ones, gets 68.9. Reconstructing the whole input clip beats reconstructing only its center frame by a wide margin, 69.6 against 63.0. MAE-ST, for its part, predicts only one of the two frames in each $2\times16\times16$ tubelet ($16\times16\times3$ values) to keep the prediction head small, and finds that sufficient.

In practice, the loss has three details that a reimplementation must get right, and that interviewers ask about. It is taken on hidden tokens only. The statistics are each tubelet's own. And the targets are normalized, never the predictions. The next exercise tests all three.

import numpy as np

def videomae_loss(pred, target_tubelets, mask, eps=1e-6):
    raise NotImplementedError

What masking saves in compute

Because the encoder reads only the visible tokens, masking cuts its cost. How much? Start from the forward cost of one Transformer layer with $T$ tokens of width $d$ (§4.7, §5.6). A FLOP is one floating-point operation, and a multiply-add counts as 2. The layer has three parts.

• The four attention projections $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$, each $d\times d$, cost $8Td^2$.
• The MLP, two matrices between width $d$ and width $4d$, costs $16Td^2$.
• The attention core, the score matrix $\mathbf{Q}\mathbf{K}^\top$ and the weighted sum of values, costs $4T^2d$.

The first two grow linearly with the number of tokens and the third with its square. That difference decides how much masking saves.

In plain words, masking 90 percent cuts the projections and the MLP by 10 times and the attention core by 100 times. For short clips the linear terms dominate, and the saving is close to 10. It approaches 100 only when the visible tokens alone far outnumber $6d$. For ViT-B ($d = 768$, so $6d = 4{,}608$) at 90 percent the saving is 13 at 1,568 tokens, 55 at 46,080, and 96 at a million.

The decoder does not shrink. It still processes all $T$ tokens, so a whole pretraining step saves less than the encoder does. Here is an exact account of MAE-ST's default model, a ViT-L encoder (24 layers of width 1,024) and a decoder of 4 layers of width 512, on 1,568 tokens with $\rho = 0.9$. Its paper reports 627.5 G "FLOPs" for a variant whose encoder also processes the mask tokens, and 81.0 G for the real one, a reduction of 7.7 times. To match it, we add three small pieces to the Transformer blocks.

• The tubelet embedding, a $1{,}536\times1{,}024$ matrix product per token (§29.2).
• The projection from the encoder width into the decoder width.
• The prediction head, $512\times768$ per token, since MAE-ST predicts one frame of each tubelet.

Two lessons come out of it. First, the paper's "FLOPs" are multiply-adds, as in every vision table of §29.3. Ratios agree with ours, while absolute numbers differ by a factor of two (§29.2 made the same check on VideoMAE's fine-tuning costs). Second, after masking, the decoder is 38 percent of the remaining work in MAE-ST's setting. For VideoMAE's ViT-B, with 160 visible tokens and a decoder of 4 layers of width 384, the decoder (37.3 GFLOPs) costs more than the encoder (28.1). That is the opening VideoMAE V2 used, later in this section.

The FLOP saving turns into a smaller wall-clock saving for three reasons. Data loading does not shrink with the mask, and §29.1 explained why decoding video is expensive. MAE-ST's remedy was to take 4 training samples, each with its own crop and mask, from every decoded video, which raised throughput 3 times. The embedding, the head, and the optimizer step cost the same as before. And shorter sequences mean smaller matrix multiplications, which keep the GPU less busy.

Two traps from §29.3 apply here. Vision tables count multiply-adds as FLOPs, so recompute one entry to learn a table's convention before comparing it with a language model's FLOPs. And test-time costs mean nothing without the number of views, the clips and crops averaged per test video. VideoMAE's 16-frame Kinetics numbers use $5\times3 = 15$ views per video, 5 clips in time times 3 crops in space (§29.1). An interviewer who asks you to estimate a pretraining budget is checking exactly these steps, the per-layer formula, the decoder that does not shrink, the multiply-add convention, and the gap between FLOPs and hours.

The widget below compares both settings with and without masking. Slide the masking ratio and watch the decoder's share of the work grow.

def mae_pretrain_flops(T, d_enc, L_enc, d_dec, L_dec, ratio, dual_ratio=0.0):
    raise NotImplementedError

VideoMAE, the whole recipe

Here is the method of Tong et al. (2022) in six steps, every piece of which we have now met.

1. Sample 16 frames, taking every fourth frame of the source video on Kinetics and every second frame on Something-Something v2 (a stride of 4 or 2). A Kinetics clip then spans 61 source frames, about 2 seconds at 30 frames per second (§29.1).
2. Embed $2\times16\times16$ tubelets into 1,568 tokens.
3. Hide 90 percent of the places with a tube mask, leaving 160 visible tokens (20 places in each of 8 slices).
4. Encode them with a plain ViT whose attention is joint over space and time. That is affordable because there are only 160 tokens.
5. Decode with 4 blocks at half the encoder's width (384 for ViT-B), and predict all 1,536 normalized pixel values of each hidden tubelet with mean squared error.
6. Pretrain for 800 epochs in the ablations (1,600 and 2,400 for the headline Kinetics and Something-Something results). Then discard the decoder and fine-tune the encoder with labels.

The production version of the pieces fits in one block. The trick, the same as in the image masked autoencoder (§28.5), is to shuffle the tokens and keep a prefix. Draw one random number per place and sort the tokens by it. The first $T_v$ tokens in that order are the visible ones, and torch.gather pulls them out. An inverse permutation, the order that undoes the shuffle, puts the decoder's input back in place.

One subtlety is worth seeing. The encoder's output is in shuffled order, and the gather with ids_restore puts every token back at its original index before the decoder adds positions. Hugging Face's implementation skips that gather. It selects tokens with the boolean mask, so its decoder sequence is the visible outputs in index order followed by the mask tokens in index order. That is a permutation of the clip, and it adds position embeddings selected in the same order. VideoMAE's own code does the same. Self-attention without an attention mask is permutation-equivariant (§4.6), meaning that reordering the input tokens reorders the outputs the same way. So only the pairing of each token with its position matters, and both versions compute the same predictions. The printed checks confirm it, and they also confirm that no gradient reaches the predictions at visible positions.

The same model through Hugging Face, with a tube mask, looks like this. It is not run here, since it downloads weights.

Results, and what data efficiency means

The results answer two questions. How well does the recipe do at each model size when it pretrains on the target dataset alone? And how does it do as the dataset shrinks, the claim in VideoMAE's title? ViT-S, ViT-L, and ViT-H below are the small, large, and huge Vision Transformers. As video models they have 22, 305, and 633 million parameters, against 87 million for ViT-B.

The phrase "data-efficient learner" in VideoMAE's title rests on one table. Every method in it pretrains on the unlabeled training clips of one dataset and then fine-tunes on the same clips with labels (ViT-B, 16 frames). UCF101 and HMDB51 are two small, older action datasets of 101 and 51 classes. The comparison method is a contrastive one of the kind in §28.2, MoCo v3 (Chen et al. 2021). It trains the network to give two augmented views of the same clip similar features and different clips different ones, and the VideoMAE authors reimplemented it for video.

Two details belong next to the table. The small datasets were pretrained with 75 percent masking for 3,200 epochs (UCF101) and 4,800 epochs (HMDB51), so "data-efficient" means few clips, not little compute. And the evaluation fine-tunes on the very data used for pretraining, which is the most favorable setting for a self-supervised method.

Why should masked modeling make good use of few clips? The authors' explanation is that reconstructing most of a clip is hard enough that it pays to learn the structure of objects and motion rather than surface statistics. A second explanation is ours, not the paper's. Every pass over a clip poses a new problem. A tube mask can be drawn in $\binom{196}{20}\approx10^{27}$ ways at VideoMAE's 90 percent, and $\binom{196}{49}\approx10^{47}$ at the 75 percent used for the small datasets. Each one asks for a different reconstruction. So the network cannot memorize answers the way a classifier memorizes labels. Both explanations are consistent with the ablations, and neither is proved.

Data quality and domain matter as much as size. Pretraining on Kinetics-400 and fine-tuning on Something-Something v2 gives 68.5, against 69.6 when pretraining on Something-Something v2 itself. Pretraining on only 42,000 Something-Something clips (68.7) beats pretraining on all 240,000 Kinetics clips (68.5) for that target (Tong et al. 2022, Table 1e and Figure 4).

Scale can compensate. MAE-ST pretrained on one million random, uncurated Instagram videos. On Something-Something v2 that matches pretraining on Kinetics-700, the 537,000-clip, 700-class version of Kinetics (73.6). It also beats every Kinetics variant on AVA, a benchmark that asks the model to find the people in movie frames and label each person's actions. In practice, pretrain on data that looks like your target before you pretrain on more data. An interviewer who hears "data-efficient" will ask how many epochs it took and what it was evaluated on, so know both answers.

Scaling up with dual masking

The compute accounting showed that once the encoder is masked, the decoder is the expensive part. For ViT-B it costs more than the encoder. The fix is to let the decoder skip some positions as well, and grade it only on the rest. That is VideoMAE V2 (Wang et al. 2023a), which used it to pretrain a video encoder with a billion parameters.

In plain words, the encoder mask should deny copies, so it hides whole tubes. The decoder mask should cover the clip, so it rotates through every place over time. The two masks have opposite goals, which is why they have opposite structure. The decoder can skip half the positions for the reason the leak analysis gave. Neighboring tokens in time carry nearly the same information, so grading half of them teaches nearly as much as grading all. VideoMAE V2 calls this the redundancy prior of video.

The cost formula reproduces V2's ViT-B numbers. Take 160 visible tokens and a decoder of 4 layers of width 384. Without decoder masking it runs on 1,568 tokens. With it, it runs on $160 + 784 = 944$. Add the embedding, the projection into the decoder, and the head. The cell below gives 35.5 and 25.8 G multiply-adds, against the reported 35.48 and 25.87.

Dual masking pays off when the decoder is a large share of the masked step. That happens at high encoder ratios with a relatively wide decoder, as for ViT-B. For ViT-g the decoder is a small share, so FLOPs fall only from 263.9 to 241.6 G. The time still falls 1.48 times, and memory from 1,753 to 1,050 MB. That suggests the saving at that scale is mostly in memory traffic rather than arithmetic. So the interview answer to "when would you mask the decoder?" is a FLOP count. Compute the decoder's share of the masked step first, and mask it only when that share is large.

What pixel reconstruction cannot learn

VideoMAE is a strong recipe for fine-tuning and a weak one for frozen features. Two tests measure frozen features (§28.6). A linear probe freezes the pretrained encoder and trains only a linear classifier on its output features. An attentive probe trains a small classifier that reads all the frozen tokens through one cross-attention layer, a fairer test for a Transformer whose information is spread over its tokens.

The usual explanation, argued by Bardes et al. (2024), is the target. Under squared error, the best prediction of a hidden tubelet is its conditional mean, the average of the possible hidden tubelets weighted by how likely each is given the visible ones (§0.4). So the network spends capacity on getting textures, edges, and lighting right on average, much of which no downstream task needs. The abstract information that tasks do need is spread through the network rather than laid out linearly at its output. Fine-tuning can reorganize it, and a frozen linear classifier cannot. This is a hypothesis, consistent with the probe results but not proved.

Three other limits belong in any honest summary. The clips are short, 16 frames covering about 2 seconds, so nothing in the objective rewards understanding minutes of video. Appearance-heavy data can be rebuilt largely from spatial context within each slice, which is one reason the masking pattern matters less on Kinetics. And tube masking stops only copies that stay in place, as the next exercise measures. In practice, this is why a team that needs a frozen video encoder, for retrieval or as the eyes of a language model, rarely uses pixel targets alone. The last part of the section shows what they use instead.

import numpy as np

def copy_baseline(tokens, mask):
    raise NotImplementedError

def shortcut_gap(tokens, ratio, rng):
    raise NotImplementedError

Masks that move with the content

The copying measurements, and Exercise 29.4e, end on a gap. A tube mask hides places, and a place keeps its content only while nothing moves. When the camera pans or an object crosses the frame, the content of a hidden place travels into places that are visible in other slices. A network that learns to follow motion can copy it from there. The remedy is to hide content instead of places, which means moving the mask with the motion.

In plain words, draw the mask once on one slice and let it ride along with the content, so whatever was hidden stays hidden wherever it goes.

Here is the gap, and its closing, measured. Take the clip of the copying experiment with the camera panning 4 pixels per frame. A time slice spans 2 frames, so the background moves 8 pixels per slice, exactly one tubelet column to the left. So the correspondence is known exactly. With base slice 0 it is $\phi_i(r, c) = (r, c + i)$ for the place in row $r$ and column $c$. For example, the content at row 3, column 2 of slice 0 sits at column 1 in slice 1 and column 0 in slice 2.

Columns past the right edge wrap around, so every slice hides the same count. A wrapped column holds new content that entered from the right. It takes the mask value of a base column whose content has already left the frame. So every piece of content, old or new, keeps one mask value for as long as it is in view. We also give the copy baseline the pan. It fills a hidden token from the nearest slice in which its displaced twin, the same content at its new place, is visible.

Under the tube mask, same-place copying solves nothing, as before. Yet the copier that follows the pan solves 0.29 of the hidden tokens. About a third of the hidden tokens have a visible displaced twin, and most of those are solved. The exceptions are mainly tokens on the moving square, which does not move with the camera. For the other two thirds, every displaced twin is hidden or has left the frame. So the tube mask had moved the shortcut, not removed it. The mask that follows the pan removes it (0.00), as the definition promises.

Real clips do not move by whole tubelets, and different parts of a scene move differently. So the published methods estimate the motion, in one of three ways. Each is named after the paper that introduced it.

• Along optical flow. Optical flow is the per-pixel motion between frames (§29.3). The method of Huang et al. (2023), called MGMAE, estimates it during pretraining with a small flow network and carries a mask drawn on the middle frame along it to both ends of the clip.
• Along codec motion vectors. A compressed video stream already stores a motion vector for each block of its predicted frames (§29.1). The method of Fan et al. (2023), called MGM, reads them for free and steers a solid block of hidden tokens onto the region that moves most.
• Learned. The method of Bandara et al. (2023), called AdaMAE, trains a small network to choose the visible tokens. It rewards choices that leave the hidden tokens hard to reconstruct.

For the two motion-guided masks the gains are larger on the motion-heavy Something-Something v2 than on Kinetics-400, most clearly at long schedules (1.5 against 0.3 points for MGMAE). That is what the copying argument predicts. The gains are 1 to 1.5 points on Something-Something v2, about the size of the gap between tube and random masking. Each paper compares against VideoMAE's published numbers with its own fine-tuning recipe. So read them as consistent evidence that masks which follow the content help, not as exact margins.

In practice, the choice is a cost question. Codec motion vectors are nearly free, flow costs pretraining time, and a learned mask adds a second network to train. An interviewer who asks about moving cameras wants to hear the definition above, mask content rather than places, and one way to get the motion.

Targets beyond pixels

The frozen-feature limit needs a different change, a new target. The field moved away from raw pixels in three directions. All three keep the masking and change what is predicted at the hidden positions.

• Hand-designed features. Histograms of oriented gradients (HOG) record, for each small cell of pixels, how strongly the edges in it point in each direction. They ignore brightness and keep local shape. MaskFeat (Wei et al. 2022) predicts the HOG of the 2D patch at the temporal center of each masked space-time cube. It masks 40 percent of cubes and feeds the mask tokens through its encoder, a multiscale video Transformer that pools tokens as it goes deeper (MViT, §29.3). So it gets no saving from an encoder that skips hidden tokens. Its MViT-L reaches 86.7 on Kinetics-400 at 352 pixels without extra data.
• Features of a teacher. Distillation trains a student network to match the outputs of a pretrained teacher. Masked Video Distillation, or MVD (Wang et al. 2023b), first trains image and video masked autoencoders, then trains a student to predict their features at the masked positions. Its ViT-L reaches 86.4 on Kinetics-400 and 76.7 on Something-Something v2. Unmasked Teacher, or UMT (Li et al. 2023), uses CLIP as the teacher. CLIP is an image encoder trained to match images with their captions, previewed in §1.8 and trained at scale in §30.2. UMT hides 80 percent of the tokens, chosen with the teacher's attention, and aligns the visible tokens' features with the teacher's. It trains a ViT-L from scratch in 6 days on 32 A100 GPUs, and it transfers to video-language tasks, which pixel targets do poorly. InternVideo2 (Wang et al. 2024) scales this to a 6-billion-parameter encoder. Its first stage aligns unmasked tokens with two teachers, the image-text model InternViT-6B and VideoMAE V2's ViT-g. Video-text contrastive learning and next-token prediction with a language model follow. Fine-tuned after the first stage alone, it reaches 92.1 on Kinetics-400.
• Features of the model itself. Instead of a fixed teacher, the network predicts the features that a slowly updated copy of its own encoder assigns to the hidden region. The copy is an exponential moving average of the encoder's weights, the same construction as in the self-distillation methods of §28.4. This is the idea behind joint-embedding predictive architectures for video, which predict in feature space rather than pixel space, so the model can ignore detail it cannot predict. V-JEPA (Bardes et al. 2024) is the model of the frozen-feature comparison above. V-JEPA 2 (Assran et al. 2025) pretrains this way on more than a million hours of video and reaches 77.3 on Something-Something v2 with a frozen encoder and an attentive probe.

In practice the choice follows the use. If you will always fine-tune, normalized pixels are cheap and strong. If the encoder will be frozen, or will feed a language model, use a feature target. That is also the short answer to the most common design question in an interview on this topic.

Masked prediction is not specific to vision. It runs through the book from hidden words in BERT (§7.1) to hidden patches (§28.5) to the hidden tubelets of this section. Masked token generators such as MaskGIT generate images by repeatedly predicting and unmasking hidden tokens (§34.4). HuBERT learns speech by predicting cluster labels of masked audio frames (§35.5). And the encoders of this section become the frame encoders of the video-language models of §29.5.

Read more

• Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Tube masking at 90 percent, the ablations, and the data-efficiency table; Appendix 6 gives the exact encoder and decoder shapes.
• Feichtenhofer et al. (2022). Masked Autoencoders As Spatiotemporal Learners. The concurrent study that prefers random masking, with the compute and wall-clock accounting of Table 1 and the experiments on uncurated data.
• Wang et al. (2023a). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. Decoder masking with running cells, and the first billion-parameter video ViT.
• Huang et al. (2023). MGMAE: Motion Guided Masking for Video Masked Autoencoding. Masks warped along optical flow from a base frame, with the ablations on base frame, warping, and flow cost.
• Wei et al. (2022). Masked Feature Prediction for Self-Supervised Visual Pre-Training. MaskFeat, which compares five targets for masked video modeling and finds HOG best.
• Li et al. (2023). Unmasked Teacher: Towards Training-Efficient Video Foundation Models. Aligning visible tokens with a CLIP teacher instead of reconstructing hidden pixels, a cheaper alternative to distilling from masked-autoencoder teachers as Masked Video Distillation does.
• Wang et al. (2024). InternVideo2: Scaling Foundation Models for Multimodal Video Understanding. Distillation, contrastive learning, and next-token prediction combined in a 6B video encoder.
• Bardes et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video. V-JEPA, with the frozen-evaluation comparison against VideoMAE in Table 5.

Watch next

▶

▶