How a stream of frames becomes a few thousand tokens, how models learn motion and time from them, and how video meets language. Built from pixels and frame rates up to the video encoders and long-video models of 2026.

This chapter is about how a model learns from video. A video is a stack of still images, called frames, shown quickly one after another. It matters now because frontier models read video every day, in search, in assistants that watch a screen or a camera, and in robots. It is also the most expensive input a model takes. The chapter follows a clip from pixels and files (this section) to tokens (§29.2), through the networks that model motion (§29.3) and the masked pretraining that trains them without labels (§29.4), to models that connect video with language (§29.5).

It rests on two ideas from earlier chapters. A Vision Transformer cuts an image into square patches and turns each patch into one token, a vector the model reads (§27.1). And the self-supervised methods of Chapter 28 learn from unlabeled images, by asking two altered copies of a photo to agree or by filling in hidden patches (§28.5). Video reuses both, with time as one more axis.

Before a model can learn from video, we need answers to three questions. How big is the stack, how is it stored on disk, and which of its frames should the model look at? Those answers set the cost of every video model in this chapter, so this section works them out with real numbers.

Here is the scale of the problem. One minute recorded on a phone at 1920 by 1080 pixels (called 1080p) and 30 frames per second saves a file of about 60 MB. Decoded into the array of pixel values a network actually reads, the same minute is 11.2 GB, almost two hundred times the file.

Now hand that minute to a Vision Transformer (Dosovitskiy et al. 2021). Its large version with 14-pixel patches, called ViT-L/14, reads a 224 by 224 image as $16\cdot16 = 256$ tokens. The minute has 1,800 frames, so it becomes 460,800 tokens. A language model's context window, the most tokens it can read at once, is often 128,000. One minute of footage overflows it more than three times.

Two facts make video affordable anyway. Consecutive frames are nearly the same picture, which is why the file is small. And a model rarely needs every frame, which is why the token count can come down. This section makes both facts precise. We compute raw sizes and bitrates, measure how much repeats from frame to frame, and see how video compressors remove the repetition. Then we choose which frames to sample, prove what sampling can miss, and end with the token bill that the rest of the chapter works to shrink.

§1.8 showed the two simple ways to embed a video, one vector per frame or one per small block of space and time. Here we start one step earlier, from the pixels and the files. All the code runs on small synthetic videos (a textured background, a moving square, a little noise) made in the first cell, because cells in the browser cannot load files.

A clip is a tensor

An image is an array of numbers with shape $H\times W\times C$, height by width by channels. A color image has $C = 3$ channels, red, green, and blue (RGB). With square patches of side $P$ pixels, a Vision Transformer makes one token per patch, so a $224\times224$ image with $P = 16$ gives $14\cdot14 = 196$ tokens (§27.1). A video adds one axis, time.

Take two seconds of video at 30 frames per second. That is 60 still images. At 224 by 224 in RGB each one holds $224\cdot224\cdot3 = 150{,}528$ numbers, so the two seconds hold $60\cdot150{,}528 \approx 9.0$ million numbers. A model sees exactly this, a block of numbers with one more axis than an image.

We write $T_f$ for the number of frames and keep $T$ for the number of tokens a model reads, as everywhere else in the book. The two-second clip above has $T_f = 60$.

Frame rates come in a few standard values. Film uses 24 frames per second and European television 25. North American television uses $30000/1001 \approx 29.97$, usually called 30, and most phones record at about 30. Sport and games use 50 or 60.

Frame index and time are not always proportional. Phones often record at a variable frame rate and slow down in dim light. So every decoded frame carries a presentation timestamp (PTS), the time at which it should be shown. Anything that cares about time, from sampling one frame per second to answering "when did the dog jump", must use the timestamps rather than $i/f_{\text{src}}$.

Our clip is grayscale, one channel holding the brightness, to keep it small. It has $24\cdot64\cdot64 = 98{,}304$ brightness values of one byte each.

How big raw video is

Start tiny. A frame of 2 by 2 pixels in RGB stores $4\cdot3 = 12$ numbers. At 8 bits each that is 96 bits. Shown 30 times per second, the stream uses $96\cdot30 = 2{,}880$ bits every second. Real video is the same multiplication with bigger numbers.

Each pixel stores a few numbers, called samples, each with $b$ bits. Usually $b = 8$. HDR (high dynamic range) video uses 10, for finer steps of brightness. The raw bitrate is pixels per frame, times samples per pixel, times bits per sample, times frames per second,

where $c_{\text{chroma}}$ is the average number of samples stored per pixel. For RGB, $c_{\text{chroma}} = 3$. Video is almost never stored as RGB, though, and the reason saves a factor of two before any clever compression starts.

Color at lower resolution

Our eyes see fine detail in brightness much better than fine detail in color. Video formats exploit this. They store brightness at full resolution and color at lower resolution, and almost nobody notices. Take a 2 by 2 block of pixels. In RGB it costs 12 numbers. Store four brightness numbers and one shared pair of color numbers instead, and it costs 6.

To do this, the video is first converted from RGB into one luma channel, the brightness, and two chroma channels, the color differences. For HD video the standard ITU-R BT.709 defines luma as $Y' = 0.2126R' + 0.7152G' + 0.0722B'$. The primes mark gamma-corrected values, the nonlinear brightness scale that cameras and screens use. The chroma channels $C_b$ and $C_r$ are scaled versions of $B' - Y'$ and $R' - Y'$. This change of basis is invertible, so nothing is lost yet. The saving comes from storing the chroma planes at lower resolution.

In plain words, in $4{:}2{:}0$ each $2\times2$ block of pixels shares one pair of color samples, so a frame costs one and a half samples per pixel instead of three. Nearly all consumer video (phones, streaming, YouTube) is $4{:}2{:}0$.

Where can a ratio near 100 come from? Recall from §0.5 that the entropy of a source is the average surprise of its symbols, measured in bits. No lossless code can use fewer bits on average than the entropy. At 8 bits, $4{:}2{:}0$ stores 12 bits per pixel, and a ratio of 93 leaves only $12/93 = 0.13$ bits per pixel.

Camera footage carries far more information than that. Sensor noise is new in every frame. Noise with a standard deviation of just one or two brightness levels out of 255 carries about 2 to 3 bits per sample on its own. So no lossless code gets anywhere near a ratio of 100, and lossless video codecs typically reach only about 2 or 3 to 1 on camera footage.

A ratio near 100 therefore needs two things. Most of each frame must be predictable from earlier frames. And the compressor must be lossy, discarding detail (including most of the noise) that the eye will not miss. The next part is about the first.

def raw_bitrate(H, W, fps, bits=8, chroma="420"):
    return H * W * 3 * fps          # placeholder: ignores chroma and bits

def compression_ratio(H, W, fps, coded_mbps, bits=8, chroma="420"):
    raise NotImplementedError

Most of a video repeats

Consecutive frames of a video are nearly the same picture, shifted a little where things moved. Start with a clip small enough to hold in your head. It has four frames of $4\times4$ grayscale pixels. All are dark except one bright pixel in row 1. That pixel moves one column to the right per frame, so in frame $i$ it sits at row 1, column $i$.

The shift is called a motion vector. Predict each frame from the previous one, then store only what the prediction got wrong. That is the whole idea of video compression. Real footage is not this tidy, so let us measure how much repeats in our synthetic clip.

The cell compares each frame with the frame lag steps later and reports three numbers.

• The mean absolute difference (MAD) of their pixels.
• The correlation of pixel values at the same position. It is 1 when one frame is an exact copy of the other up to brightness and contrast, and near 0 when they are unrelated (§0.4).
• The best correlation after shifting one frame against the other by up to 8 pixels, which is what a motion vector would supply.

A clip of independent random noise is the control.

Two things stand out. First, the correlation at the same pixel falls fast, from 0.80 at lag 1 to 0.09 at lag 4. The camera pans one pixel per frame, and the texture changes within a few pixels, so the same position soon shows different content.

Second, after the best shift the correlation stays near 0.7 even 8 frames apart. The content has not changed, it has only moved. It stays below 1 because the moving square goes its own way, and no single shift aligns it together with the background. Noise has no repetition to find at any lag.

That is why compressors search for motion instead of subtracting frames. It is also why masked video modeling, which trains a model to fill in hidden patches of a clip, must stop it from copying a hidden patch from the next frame.

I, P, and B frames

A codec (coder and decoder) is the program that compresses video and restores it. The most widely deployed one is H.264, also called AVC. Its successors HEVC, VP9, and AV1 share its vocabulary. They all turn the observation above into a file format.

B-frames change the order of work. A B-frame needs its future reference first, so the decoder must receive and decode that later frame before the B-frame. Take the display order I0 B1 B2 P3 B4 B5 P6, where B1 and B2 refer to I0 and P3. The frames are stored and decoded in the order I0 P3 B1 B2 P6 B4 B5.

So a video file stores two timestamps per frame, one for decoding and one for display. That is also why "frame number" in a stream can mean two different things. Always ask which order a tool counts in.

In plain words, each block looks around the previous frame for the patch that matches it best, remembers where that patch was, and keeps only the leftover difference.

Watch the sign. Under this convention the moving pixel of the hand example has motion vector $(0, -1)$, because the pixel now at column $x$ came from column $x - 1$. The pixel moved right, and its vector points left, back to where it came from. The sign trips up many implementations, and Exercise 29.1d checks it.

The residual is compressed the way JPEG compresses an image, in three steps.

1. It is cut into small blocks, and each block goes through a discrete cosine transform (DCT). The DCT rewrites the block as a weighted sum of cosine patterns, from smooth to fine.
2. The weights are divided by a step size and rounded, which is called quantization. This rounding is where information is lost.
3. The rounded weights are entropy coded, which gives frequent values short codes.

A good prediction leaves a residual close to zero. Then most weights round to zero and cost almost nothing.

Before writing block matching yourself, test the idea in its crudest form, one motion vector for the whole frame. On a pure camera pan it should work. With an object that moves on its own, it should not.

On the pure pan, the vector $(0, 1)$ cuts the residual energy to 5 percent of plain differencing, a twentyfold drop. What remains, $2.0\times10^{-4}$, is set by noise. The first cell added Gaussian sensor noise of standard deviation $\sigma = 0.01$ to every frame. After a perfect shift, the residual is the difference of two independent noise images, so its mean square is $2\sigma^2 = 2\times10^{-4}$.

That floor matters in practice. Sensor noise is new in every frame, so no prediction can remove it, and it sets the least a codec must spend. With the moving square, the best single vector still leaves 63 percent, because the square moves differently from the background. Hence blocks, each with its own vector.

The widget below runs block matching on two frames. Like a real codec, it lets a border block take its source from just past the frame's edge, filled by repeating the edge pixels. Choose the kind of motion, the block size, and the search radius, and compare the plain difference with the residual.

Exhaustive search is expensive. Each block tries $(2r+1)^2$ vectors at $n_b^2$ absolute differences each, so a frame costs $HW(2r+1)^2$ of them whatever the block size. At 1080p with $r = 16$ that is $1920\cdot1080\cdot33^2 \approx 2.3\times10^9$ per frame, or $6.8\times10^{10}$ per second at 30 frames per second. Real encoders therefore use fast searches that start from the vectors of neighboring blocks.

import numpy as np

def block_motion(prev, cur, block=8, radius=4):
    H, W = cur.shape
    return np.zeros((H // block, W // block, 2), dtype=int)   # placeholder: no motion

def compensate(prev, mv, block):
    return prev.copy()                                           # placeholder: copy the frame

Codec motion vectors are not the true motion of the scene. They are whatever block copy makes the residual cheapest. On flat walls they can point anywhere, and at object edges they are blocky. The measurement of true motion is optical flow, a motion vector for every pixel estimated from how brightness moves. §29.3 derives it, along with the two-stream networks built on it.

Still, codec vectors come free once a video is compressed. An action recognizer can be trained directly on the I-frames, motion vectors, and residuals stored in the compressed file, which is faster than decoding everything to pixels (Wu et al. 2018).

Why reading video is the slow part of training

Compression has a cost on the reading side. To decode a P-frame you need its reference frame, which needs its own reference, and so on back to the last I-frame. So reaching an arbitrary frame means decoding forward from the previous I-frame.

Decoding runs on CPUs by default, and a fast GPU can consume clips faster than the CPUs decode them. Feichtenhofer et al. (2022) measured this. Their masked autoencoder for video (§29.4) runs its encoder only on the visible tenth of the tokens.

The other standard fixes are these.

• Re-encode the training set with short GOPs and a modest resolution.
• Store pre-extracted frames as JPEG images, which costs more disk and gives cheap random access.
• Decode on the GPU with its hardware decoder (NVDEC on NVIDIA GPUs).
• Seek to the I-frame just before the clip instead of decoding from the start of the file.

The code below shows two real decoders. PyAV wraps FFmpeg, the standard open-source video library. TorchCodec is PyTorch's own decoding library, and it can decode on the GPU.

Many older codebases call torchvision.io.read_video (with pts_unit="sec"). torchvision deprecated its video decoding in version 0.22 and removed it in version 0.26, in favor of TorchCodec, so new code uses TorchCodec.

Choosing which frames to look at

A model cannot read every frame of every video, so each system chooses which frames to keep. That choice decides what the model can possibly see, so it is part of the model. The right choice depends on the task. In video research the tasks are fixed by a handful of public benchmarks, and every accuracy in this chapter is measured on one of them, so we name them once here.

Kinetics and Something-Something v2 are the pair to remember. Many Kinetics classes can be named from one frame by the scene or the objects, so a model that ignores time still scores well. Something-Something v2 cannot be solved without motion. §29.3 uses this contrast to test whether an architecture really models time.

Now the samplers. Each one takes a video and returns the frame indices the model will see.

The strategies trade fine detail in time against coverage of the whole video.

• A dense clip of 16 frames at 30 frames per second covers 0.53 s in fine detail, enough to see a hand rotate a cup.
• VideoMAE, the masked autoencoder for video of §29.4 (Tong et al. 2022), pretrains on Kinetics with $16\times4$ clips. Each spans $15\cdot4+1 = 61$ source frames, about 2 s at 30 frames per second and 2.4 s at 25. On Something-Something v2 it pretrains with stride 2 and fine-tunes with segment sampling of 16 segments. So does MViT, a vision Transformer that shrinks its token grid and widens its channels stage by stage (Fan et al. 2021).
• Segment sampling with $K = 3$, the TSN default, covers the whole video with three frames and almost no motion detail.
• Fixed-rate sampling at 1 frame per second, the default of the Gemini API, reads a long video evenly. That is the right default for answering questions about a video and the wrong one for anything that lasts less than a second.

An interviewer who asks which sampler to use wants the task first. Fine motion needs frames a fraction of a second apart, and a question about a whole video needs coverage.

import numpy as np

def sample_frames(num_frames, n, mode, rng=None, stride=1, src_fps=None, target_fps=None):
    raise NotImplementedError

Views, and honest test-time cost

A classifier trained on short clips must still label a whole video at test time. The standard protocol runs it on several clips spread over the video and several crops of each frame, then averages the scores. Each clip-and-crop pair is called a view.

So compare models by total cost per video at their reported view counts, and count views as part of the cost when you design a system. A model that needs 30 views to reach its accuracy is 30 times more expensive than its GFLOPs column says. An interviewer who shows you a table of GFLOPs will expect you to ask for the views.

What sampling can miss

Sampling at one frame per second is cheap, but it can miss things. Suppose a goal, a fall, or a flash of a license plate lasts half a second, and we sample once per second. Whether a sampled frame lands inside the event depends on where the sampling instants fall relative to it. If the first instant is equally likely to be anywhere in the first second, half of those positions put an instant inside the event and half do not. The event is caught half the time.

The same reasoning works for any event length and any rate. The model is a sampling grid with a uniformly random offset. It is the right model when we know nothing about where events happen.

In plain words, each sampled frame stands for $1/f_{\text{smp}}$ seconds. An event shorter than that is seen with probability equal to the fraction of a sampling period it lasts. At 1 frame per second a half-second event is missed half the time. At 2 frames per second a 0.3 s event is caught with probability 0.6.

Seeing motion needs at least two frames of the event. The theorem with $k = 2$ says that a 1.5 s action at 1 frame per second appears in two frames only half the time. The cell checks the theorem by simulation.

The theorem assumes the offset is random relative to the event. A fixed sampler applied to a fixed video is deterministic, so "random" here means averaged over where events happen. That is the right average for a benchmark or a product.

It says that a sampler's blind spots are not rare accidents. Among events of $\delta < 1/f_{\text{smp}}$ seconds, a fraction $1 - \delta f_{\text{smp}}$ is never seen. An interviewer who asks whether 1 frame per second is enough for a video product wants this number.

The next widget puts the samplers and the capture probability side by side, with the token count each choice costs. Drag the shaded event along the timeline to see whether any sampled frame shows it. The strip below the timeline marks every place the event could start, green where some frame would catch it, so its green share is the capture probability.

Temporal aliasing

Short events can be missed. Fast repeating motion is worse, because it is not missed but misreported. This is aliasing, one motion appearing as a different one after sampling.

A tone sampled too slowly shows up as a lower tone, below half the sampling rate, as §35.1 works out for sound. A spoked wheel does the same. A wheel with $n_{\text{sp}}$ identical spokes looks the same after $1/n_{\text{sp}}$ of a turn, so what a camera sees of it repeats $n_{\text{sp}}$ times per revolution. Sampled by frames, that pattern is misread as a slower one, possibly running the other way. For rotation the sign matters, because the eye reads a small backward step as backward motion. Here is the signed version.

In plain words, the frames cannot distinguish a turn of $\theta$ from a turn of $\theta$ minus one whole spoke gap. The eye picks whichever step is smaller, and when the true step is just under a gap, the smaller step points backward.

The illusion is named after the wagon wheels of old Western films, so here is the theorem on one of them.

The widget below films such a wheel. Keep 12 spokes at 24 frames per second. Before you raise the rotation from 0, predict the rate at which the wheel first seems to turn backward and the rate at which it stands still. Then set the stagecoach's 1.9 revolutions per second and lower the spokes to 1. Ask why the illusion vanishes, although the wheel turns exactly as fast. Finally, mark one spoke, play the wheel again, and compare the motion of the marked spoke with that of the others.

The widget shows the theorem at work. With $n_{\text{sp}} = 12$ and $f_{\text{smp}} = 24$ the bound is $24/24 = 1$ revolution per second.

• Just above 1 revolution per second the wheel seems to turn backward at almost the same speed. The backward turn slows as the true rate rises toward 2.
• At 2 revolutions per second the step is one whole spoke gap, and the wheel freezes.
• At exactly 1 revolution per second the step is half a gap. Forward and backward give the same pictures, and the direction is undefined.
• With one spoke the picture repeats only after a full revolution. The bound rises to 12 revolutions per second, and 1.9 is seen as it is.

A model faces the same limit as the eye. A clip sampled at $f_{\text{smp}}$ frames per second cannot represent repeating motion faster than $f_{\text{smp}}/2$ cycles per second. Propellers, fans, vibrating strings, and the fine timing of a golf swing need dense sampling or a high source frame rate. A model trained on sparsely sampled frames can learn the aliased motion as if it were real.

import numpy as np

def apparent_spin(rps, spokes, fps):
    return rps                       # placeholder: no aliasing at all

def capture_probability(delta, fps, k=1):
    return delta * fps               # placeholder: can exceed 1, ignores k

The token bill

Now the arithmetic the whole chapter has been building toward. Suppose each sampled frame is resized and cut into patches on its own, and a model then reads all the frame tokens together.

The general count multiplies the same factors.

The formula just multiplies the definitions together. What matters is how $T$ grows. Tokens grow linearly with duration, frame rate, and pixel area, and shrink by $m^2$ with merging.

Attention has two terms. The first, $8Td^2$, is the four projections (queries, keys, values, and output), and it grows linearly with $T$. The second, $4T^2d$, is the scores and the weighted sum of values. It grows with the square of everything that grows $T$, so doubling the frame rate quadruples it.

Real systems rarely run one attention over every patch of every frame, and §29.3 derives the cheaper factorizations. But in today's video language models the frame tokens end up side by side in a language model's context. So the count $T$ is exactly what the context window pays.

The codec's idea can also run inside the model. Run-length tokenization drops each patch that repeats the patch at the same place in the previous frame. On long, mostly static videos it removes most of the tokens (Choudhury et al. 2024).

import math

def video_tokens(duration_s, fps, H, W, patch, merge=1, audio_tps=0, per_frame_overhead=0):
    return int(duration_s * fps * math.ceil(H / patch) * math.ceil(W / patch) / merge)

def max_fps_for_budget(duration_s, H, W, patch, budget, reserve=0, fps_grid=(0.25, 0.5, 1, 2, 4, 8),
                       merge=1, audio_tps=0, per_frame_overhead=0):
    raise NotImplementedError

Augmenting a clip

Data augmentation means transforming each training example at random every time it is read, so the model never sees exactly the same input twice. Video models need it badly. Kinetics-400's 240,000 training clips are few by the standards of image pretraining, and video models overfit them easily. Test clips, by contrast, are transformed in one fixed way.

The main video models share one recipe. They include I3D, a 3D convolutional network made by extending a trained image network through time (Carreira and Zisserman 2017). The others are the SlowFast and X3D networks of the views example, MViT (Fan et al. 2021), and VideoMAE.

• Temporal jitter. The start $t_0$ of a training clip is drawn uniformly over the valid starts on every read, as in the strided sampler above. So each epoch sees a different window of each video.
• Spatial jitter. Resize each frame so that its short side is a random length, for example in $[256, 320]$ pixels (SlowFast and X3D; I3D fixes it at 256). Take a random $224\times224$ crop, and flip left to right with probability one half.
• Test transforms. Resize the short side to a fixed length (256 for SlowFast, X3D, and MViT). Take either the center crop or three crops that together cover the long side. These are the spatial crops of the test views defined above.
• Stronger regularizers for Transformers. VideoMAE's fine-tuning adds the four below. It also uses stochastic depth (randomly skipping whole residual blocks during training), dropout, and repeated augmentation, which puts two differently augmented copies of each clip in the same batch.
  • RandAugment applies randomly chosen image operations, such as rotation, shear, and contrast changes, at a shared strength (Cubuk et al. 2020).
  • Mixup blends two clips and their labels with a random weight (Zhang et al. 2018).
  • CutMix pastes a box from one clip into another and mixes the labels in proportion to the box's area (Yun et al. 2019).
  • Label smoothing moves a little of the target probability from the true class to the others (§0.5).

Two rules are specific to video. Both are correctness rules, not tuning choices.

The cell below makes both rules visible. It takes a pure camera pan, one pixel per frame, and crops it to $56\times56$ in three ways. Then it estimates each step's motion with best_global_shift from earlier, the function that finds the single shift best predicting one frame from the previous one.

The shared crop recovers the pan's motion vector, $(0, +1)$ at every step, because content moving one pixel left comes from one column to the right. The per-frame crops report steps such as $(-2, +4)$ and $(+6, 0)$. Each is the change of crop position plus the true step, so the jitter, not the scene, now dominates the motion signal. A model trained on such clips learns that motion is noise.

The flipped clip moves at $(0, -1)$, the same pan in the opposite direction. That is harmless for "playing guitar" and changes the answer for "pushing something from left to right". An interviewer who asks how you would augment Something-Something v2 is checking for exactly this.

In PyTorch

In PyTorch a decoded clip arrives as uint8, one byte per sample. The block below does four things.

• It converts the clip to the $(B, C, T_f, H, W)$ layout of nn.Conv3d.
• It normalizes with the ImageNet statistics that most video models inherit from their image backbones.
• It uses Tensor.unfold to cut a long video into overlapping strided clips without copying memory.
• It draws one training clip with temporal jitter and one spatial transform shared by all its frames.

The scales are the standard recipe halved for 112-pixel frames, a short side in $[128, 160]$ and a crop of 112.

The last line is a preview. A 3D convolution whose stride equals its kernel size cuts each clip into non-overlapping space-time blocks, the tubelets of §29.2.

Read more

• Wiegand et al. (2003). Overview of the H.264/AVC video coding standard. The design of the most widely deployed codec, from macroblocks and motion compensation to the integer transform and entropy coding. The standard itself is ITU-T H.264.
• Wu et al. (2018). Compressed Video Action Recognition. A clear introduction to I and P frames, motion vectors, and residuals, and a model that learns from them directly.
• Wang et al. (2016). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. Segment sampling and its testing protocol.
• Goyal et al. (2017). The "something something" video database for learning and evaluating visual common sense. The motion-defined benchmark, and why its classes are templates over objects.
• Kay et al. (2017). The Kinetics Human Action Video Dataset. 400 classes, at least 400 clips each of about 10 seconds, 306,245 clips in the first release, the benchmark most video numbers refer to.
• Feichtenhofer (2020). X3D: Expanding Architectures for Efficient Video Recognition. Its Kinetics-400 table, with GFLOPs times views for every model, is the source of the cost comparison in this section.
• Feichtenhofer et al. (2022). Masked Autoencoders As Spatiotemporal Learners. Section 5.1 and Table 1 measure how data loading becomes the bottleneck once computation is cheap.
• Choudhury et al. (2024). Don't Look Twice: Faster Video Transformers with Run-Length Tokenization. The codec's insight applied to tokens, removing patches that repeat over time.

Watch next

▶

▶

▶