This section is about how a video network combines what it sees inside each frame with how things change from one frame to the next. A single frame shows what is in a scene. Only the change between frames, the motion, shows what is happening. Every video architecture is one answer to the same design question. Where in the network, and how, does information from one frame reach another?

The answer matters for two reasons. The same network can score well on one video benchmark while using no motion at all, so a gain means little until you know what the benchmark measures. And video inputs are so large that a poor choice can cost ten times the compute for no gain in accuracy.

Start with a concrete test. Take a strong image model and run it on every frame of a clip, never letting one frame look at another. Average its outputs over the frames. How far behind a real video model is it? Two benchmarks give very different answers. Kinetics-400 is the most used action-recognition benchmark, with 400 human actions in YouTube clips of about 10 seconds (Kay et al. 2017). Something-Something v2 has 174 actions acted out with everyday objects, named by templates such as "Dropping something into something" (Goyal et al. 2017). Both report top-1 accuracy, the percentage of test clips whose highest-scoring class is the right one.

On Kinetics the frame-by-frame model is only 1.1 points behind, because a swimming pool or a guitar already names most of its actions. On Something-Something it is 22.9 points behind. "Pushing something left" and "pushing something right" look the same in any single frame. So the first lesson of this section is about data, not networks. The benchmark decides whether time matters. Read every table below with that in mind, because an interviewer who hears "our model gained a point on Kinetics" will ask how you know it uses motion at all.

There are two broad ways to give a network motion. You can make motion explicit, by measuring how each pixel moves and feeding those measurements to a network. Or you can make it implicit, by letting the network's filters or its attention look across frames. The section follows both, in roughly the order they were invented.

1. Image networks that combine frames inside themselves, and why they did little better than a single frame.
2. Optical flow, the measured motion of every pixel, and the networks built on it.
3. 3D convolutions, and cheap ways to add time to 2D networks.
4. Video Transformers and their attention patterns, with every cost derived and published compute numbers reproduced to within 1 percent.
5. The two designs that 2026 systems use, and why.

This path is one stretch of a thread that runs through the book. Image models began with convolutions, small learned filters slid over the image (§0.8). Sequence models began with recurrence, a state updated one step at a time (§3.3). Both moved to attention, which lets every position read from every other position in one step (§4.7). Vision followed once images were cut into patches (§27.1), at the price of the locality and weight sharing that convolutions build in (§27.3). Video went through the same three stages, and at the end of this section recurrence comes back.

A little notation is shared by the whole section. A clip has $T_f$ frames. A video Transformer cuts the clip into tubelets, small blocks a few frames deep and a few pixels on a side, and turns each one into a token (§29.2). That gives $n_t$ tokens along time and $n_s = n_hn_w$ tokens in each time slice ($n_h$ rows by $n_w$ columns), so $T = n_tn_s$ tokens in all. A time slice is the set of $n_s$ tokens that share one position in time. The model width, the length of each token vector, is $d$.

FLOPs, floating-point operations, follow the book's convention. Multiplying an $m\times k$ matrix by a $k\times n$ matrix costs $2mkn$ FLOPs, one multiply and one add per term (§4.7). A GFLOP is a billion of them. Most video papers count one multiply-add as one FLOP instead, half our count, and the part of the section on reproducing published numbers shows how to convert.

Where to combine frames in an image network

The first large study of deep video models asked the simplest possible question (Karpathy et al. 2014). Take a ConvNet, a stack of small learned filters slid over an image (§0.8), that classifies one image. Where inside it should the frames of a clip be combined? There are four natural answers.

The paper's numbers make this concrete. Early fusion used $T_f = 10$ frames, about a third of a second, so its first layer had $11\times11$ kernels over $10\cdot3 = 30$ channels. Slow fusion applied filters spanning 4 frames at stride 2 to the 10 frames, which gives 4 outputs in time. The next two layers used filters spanning 2 frames at stride 2, so the third convolution sees all 10 frames.

Both designs are special cases of the 3D convolution defined later in this section, a filter that spans several frames as well as a patch of pixels. Early fusion is one such filter that spans the whole clip in time. Its $k\times k$ kernel over $T_fC$ channels has $k^2T_fCF$ weights for $F$ filters, exactly as many as a kernel $T_f$ frames deep. Slow fusion is a stack of them with a temporal stride.

Two things held fusion back. The first is the data. A sport is usually recognizable from one frame by its field, equipment, and clothing, the same reliance on appearance the opening found in Kinetics. The second is the architecture. Filters that must discover motion from raw pixels, trained from scratch, learned little that a single frame did not already show.

A recurrent network on top of per-frame features

A second family kept the image network intact and put time on top of it. Run a pretrained 2D ConvNet on every frame and take one feature vector per frame from its last layer. Then feed the sequence of vectors to an LSTM (Donahue et al. 2015, Ng et al. 2015). An LSTM is a recurrent network that reads one vector per step. It keeps a hidden state, a summary of what it has seen so far, and learned gates decide how that state changes at each step (§0.8, §3.3).

The gains were small. The cleanest comparison gave a ConvNet plus LSTM and a 3D network the same ImageNet-pretrained backbone. The LSTM model reached 63.3 percent on Kinetics-400, against 71.1 for the 3D network on the same RGB frames (Carreira and Zisserman 2017).

Three reasons explain the gap.

• The LSTM reads features from the last layer, after the backbone has looked at each frame alone. Fine motion is gone before time is modeled at all.
• It is sequential. Training backpropagates through time, one step after another, so the frames of a clip cannot be processed in parallel.
• Only the backbone is pretrained. The recurrent layer starts from random weights on small video datasets.

Every design so far has one of two flaws. In late fusion and the recurrent models, no convolution ever sees two frames. In early and slow fusion, the filters that do see several frames must learn motion from raw pixels with no pretraining. The rest of the section fixes both, first by handing the network motion directly, then by letting pretrained filters look across time.

Optical flow makes motion explicit

Look at one pixel of a video. If the scene moves, the brightness that was at that pixel in one frame shows up somewhere nearby in the next. Optical flow is that displacement, one arrow per pixel, measured in pixels per frame. Computing it first, and handing it to a network, was the first way deep video models got motion.

Here is how a single pixel can measure its arrow. Take a row of pixels whose brightness rises by 2 for every pixel to the right, reading 10, 12, 14, 16, and so on. Between two frames the pattern slides 2 pixels to the right. A pixel that read 14 now shows what was 2 pixels to its left, which is 10. Its brightness changed by $-4$. That change is minus the slope, 2, times the shift, 2. So a pixel that knows its slope and its change in time can solve for the motion, $u = 4/2 = 2$ pixels per frame.

The ramp example used a first-order version of this equation. A Taylor expansion turns it into a precise statement and bounds the error.

In plain words, the change in brightness at a fixed pixel ($I_t$) must be explained by the pattern sliding past it. That change is the slope times how far the pattern slid, plus a curvature term that grows with the square of the motion. When the motion is large, that term is not small. The Lucas-Kanade cell below deals with it by warping, shifting the second frame back by the current estimate of the motion, and then solving again for the small motion that remains.

Write the constraint as $\nabla I\cdot(u, v) = -I_t$, with $\nabla I = (I_x, I_y)$ the brightness gradient, the direction in which brightness rises fastest. The constraint fixes only the part of the flow along the gradient, which is $-I_t/\lVert\nabla I\rVert$. It says nothing about the part along the edge, perpendicular to the gradient. This is the aperture problem. Seen through a small opening, an edge sliding along itself looks the same as an edge standing still, and every single pixel has this problem.

Solving for the flow in a small window

One equation in two unknowns is not enough. Lucas and Kanade (1981) fixed that by assuming every pixel in a small window around a point moves the same way. A window of $5\times5$ to $15\times15$ pixels gives 25 to 225 equations for the same two unknowns, and least squares solves them (§0.10).

Two pixels already suffice when they see different edges. Say one sits on a vertical edge with $I_x = 2$, $I_y = 0$, $I_t = -4$. The other sits on a horizontal edge with $I_x = 0$, $I_y = 3$, $I_t = -3$. The first says $2u = 4$, so $u = 2$. The second says $3v = 3$, so $v = 1$. Together they pin the motion down to $(2, 1)$. If both pixels sat on vertical edges, $v$ would stay unknown however many pixels you added.

In plain words, Lucas-Kanade picks the one motion that best satisfies every pixel's constraint in the window. The structure tensor sums, over the window, how strongly brightness changes in each direction. In the two-pixel example $\mathbf{M}_{\text{st}} = \begin{bmatrix} 4 & 0 \\ 0 & 9\end{bmatrix}$, so $\lambda_2 = 4$ and the window is well posed. With two vertical edges it would be $\begin{bmatrix} 8 & 0 \\ 0 & 0\end{bmatrix}$, with $\lambda_2 = 0$.

In plain words, a window sees motion in every direction its gradients point. A corner or a textured patch has gradients in two directions and pins the flow down. An edge has gradients in one direction, and the motion along it is invisible. The smaller eigenvalue $\lambda_2$ measures how well posed the window is. That is why Shi and Tomasi (1994) used it to score which points of an image are good to track.

In the widget, each pixel of the window draws its own constraint line, the set of motions $(u, v)$ it allows. The estimate is the motion that best satisfies all the lines at once, in the least-squares sense, so pixels with strong gradients count more.

Here is the method from scratch, on a smooth blob that moves half a pixel right and one pixel down. The spatial derivatives are central differences, the difference of the two neighbors divided by 2, taken on the average of the two frames. $I_t$ is the difference of the frames. That way all three derivatives are measured at the same moment, halfway between the frames.

The practical Lucas-Kanade method runs this iteration on an image pyramid, copies of the frames at halved and quartered resolution. It starts at the coarsest level, where a large motion is only a pixel or two, and refines the estimate level by level. The next exercise computes the flow at every pixel at once, a common interview task because it tests vectorization as much as the math.

import numpy as np

def lucas_kanade_dense(I0, I1, win=7, tau=1e-3):
    raise NotImplementedError

Flow from Lucas-Kanade is fragile, because each of its assumptions fails somewhere.

• Brightness constancy fails under shadows and reflections.
• Small motion fails for fast objects, which pyramids fix only up to a point.
• One motion per window fails at object boundaries and where one object hides another.
• Texture fails on flat walls, where $\lambda_2 \approx 0$.

Two-stream networks feed flow to a ConvNet

In 2014, ConvNets trained on stacks of raw frames were at most 1.6 points better than single-frame ones, as the fusion results showed. Simonyan and Zisserman (2014) separated appearance from motion and gave each its own network.

Why did a precomputed input help so much? A flow field does not change with clothing, lighting, or background. So the temporal stream learns motion patterns from very little data, and UCF-101 has only about 13,000 clips. The price was computing flow for every frame offline. That step dominated the cost of the whole system and capped the frame rate a model could afford. The rest of this section is the story of learning motion from pixels instead.

3D convolutions learn motion from pixels

The direct way to let a network see motion is to give its filters a time axis. Recall from §0.8 what a 2D convolution is. With $C$ input channels, $F$ filters, and a $k\times k$ kernel, it applies the same small dense layer to every $k\times k\times C$ patch of its input, so it has $k^2CF$ weights. A $3\times3$ convolution from 64 to 64 channels has $9\cdot64\cdot64 = 36{,}864$ weights. Let each filter also span 3 frames and it has $3\cdot36{,}864 = 110{,}592$, three times as many, and each output now depends on three frames.

In plain words, each output looks at a small box of the clip, $k_t$ frames deep and $k\times k$ pixels across, and takes a learned weighted sum of everything in it. The tubelet embedding of §29.2 is the special case whose stride equals its kernel, so the boxes do not overlap. Here the stride is 1, the boxes overlap, and the layers stack. A network's view therefore grows in time exactly as it grows in space.

Tran et al. (2015) built C3D, the first widely used 3D network. It has eight $3\times3\times3$ convolution layers and reads 16-frame clips cropped to $112\times112$ pixels, a kernel shape chosen because it beat the alternatives in their experiments. Trained from scratch on Sports-1M, it was shallow and low in resolution next to the image networks of its day. There was no labeled video dataset the size of ImageNet, and no way to start from image weights.

Carreira and Zisserman (2017) removed both obstacles. Their inflated 3D ConvNet, I3D, starts from an image network trained on ImageNet's 1.3 million labeled photos. It turns every $k\times k$ filter into a $k\times k\times k$ one by repeating the 2D filter $k$ times along time and dividing by $k$. Feed it a video in which every frame is the same image. Each 3D filter then sums $k$ identical copies divided by $k$, so the network computes exactly what the image network did (§29.2 proves this for a whole network). The same paper released Kinetics, the first video dataset large enough to train such a network. Together, inflation and Kinetics made 3D ConvNets work.

In practice, inflation is still how a video model starts from an image model, and an interviewer will often ask you to show why the inflated network reproduces the image network on a static clip.

Attention arrives as a non-local block

A convolution sees only a small neighborhood. Wang et al. (2018) added a non-local block that lets every position of a feature map read from every other position in space and time,

passed through one more learned linear map and added back to the input through a residual connection. Here $\mathbf{x}_i$ is the feature vector at position $i$, $f$ scores how related positions $i$ and $j$ are, $g$ is a learned linear map, and $\mathcal{C}(\mathbf{x})$ normalizes. Choose $f(\mathbf{x}_i, \mathbf{x}_j) = e^{\theta(\mathbf{x}_i)^\top\phi(\mathbf{x}_j)}$ for learned linear maps $\theta, \phi$, and $\mathcal{C}(\mathbf{x}) = \sum_j f(\mathbf{x}_i, \mathbf{x}_j)$. Then the block is exactly single-head self-attention without the $1/\sqrt{d}$ scale (§4.3), with $\theta$ making queries, $\phi$ keys, and $g$ values.

The baseline was a ResNet-50, a 50-layer ConvNet with residual connections (§0.8), applied with 2D filters to the frames of a clip. On Kinetics, five non-local blocks raised it from 71.8 to 73.8 percent when they attended over space and time. Attending over space only gave 72.9, and over time only 73.1. Added to an I3D, the blocks raised it from 73.3 to 74.9. Attention across space and time helped even on top of 3D convolution, a hint of what came next.

Splitting a 3D convolution into space, then time

A $3\times3\times3$ kernel mixes space and time in one linear map. Tran et al. (2018) asked whether that is necessary. Their R(2+1)D block splits it in two. First comes a spatial $1\times k\times k$ convolution into $C_{\text{mid}}$ channels, then a nonlinearity, then a temporal $k_t\times1\times1$ convolution. They chose the intermediate width $C_{\text{mid}}$ so that the pair has the same number of weights as the 3D kernel it replaces.

Proof. The spatial convolution has $k^2CC_{\text{mid}}$ weights and the temporal one $k_tC_{\text{mid}}F$, a total of $C_{\text{mid}}(k^2C + k_tF)$. Setting it equal to $k_tk^2CF$ gives the ratio. Rounding down lowers $C_{\text{mid}}$ by less than one, which lowers the count by less than $k^2C + k_tF$. $\blacksquare$ (The paper writes $d$ for the spatial kernel size and $N_{i-1}, N_i$ for the channels. We keep $d$ for the model width.)

At equal parameters the factorized block has twice as many nonlinearities. Tran et al. found it reached lower training error as well as lower test error than a full 3D network of the same depth, which they read as easier optimization. Trained from scratch on Kinetics RGB frames, R(2+1)D-34 reached 72.0 percent, against 67.5 for I3D under the same conditions (their Table 5). The same split, a spatial operator followed by a temporal one, returns below as divided attention.

def conv3d_params(k_t, k, C, F):
    raise NotImplementedError

def r2plus1d_mid_channels(k_t, k, C, F):
    raise NotImplementedError

def r2plus1d_params(k_t, k, C, F):
    raise NotImplementedError

In PyTorch each block is a few lines. The cell builds both, then checks that they have the same number of weights and the same output shape.

The same blocks ship in torchvision, which also has the MViTv2 model met later in this section.

Adding time to a 2D network cheaply

3D networks are heavy, since every filter costs $k_t$ times as much as its 2D version. Two ideas get much of the benefit of looking across frames at a fraction of that cost.

The first covers a long video cheaply. Temporal Segment Networks (Wang et al. 2016) split a video into a few equal segments and run a 2D network on one frame sampled from each (the segment sampling of §29.1). The video's score is the average of the per-frame scores. That sees the whole video, but it models no motion beyond what single frames show.

The second adds motion for free. Take a 2D network that processes the frames of a clip together, as a batch. Inside it, move a few feature channels one frame forward in time and a few one frame back, then let the next convolution do its usual work. Here is a tiny case. Say a layer has 8 channels per frame. Channel 0 of frame $t$ is replaced by channel 0 of frame $t + 1$, channel 1 by channel 1 of frame $t - 1$, and channels 2 to 7 stay put. The next convolution mixes all 8 channels at each pixel, so each of its outputs now combines features of frames $t - 1$, $t$, and $t + 1$. This is the temporal shift module, TSM (Lin et al. 2019).

Each block now sees 3 consecutive frames. Stacking blocks widens that reach with depth, as stacked convolutions widen their view in space. In fact a shift followed by a $1\times1$ convolution is exactly a temporal convolution with a kernel 3 frames long. The kernel is restricted so that each input channel uses only one of its three taps. Exercise 29.3e proves this by building that kernel. The restriction is why the shift costs nothing extra.

That table is the opening lesson again. Something-Something cannot be solved without motion, and Kinetics mostly can. In practice, TSM is the first thing to try when you have a good 2D network and need a cheap or streaming video model.

import numpy as np

def temporal_shift(X, frac=1/8, online=False):
    raise NotImplementedError

def tsm_as_temporal_conv(W, frac=1/8):
    raise NotImplementedError

Two frame rates at once

What is in a scene changes slowly, and how it moves changes fast. A person stays a person for the whole clip, while their hand may cross the frame in a tenth of a second. Feichtenhofer et al. (2019) built that observation into SlowFast, a network with two pathways that read the same clip at two frame rates.

The paper reports that the Fast pathway takes about 20 percent of the total compute in practice. That is more than the one-eighth estimate, because every Fast block has a temporal kernel of size 3, while the early Slow blocks are purely 2D. In plain words, the Fast pathway is a thin network that watches closely, and the Slow pathway is a wide one that glances.

Published costs need one more term before the numbers make sense. A view is one crop of a test video in space and time. The model scores each view, and the video's score is the average over views (§29.1). So the cost of classifying one video is the cost per view times the number of views, and papers report both.

X3D grows a tiny 2D network one axis at a time. The axes are frames, frame rate, resolution, width, bottleneck width, and depth, and at each step it keeps the expansion that buys the most accuracy per FLOP. X3D-M matches the smaller SlowFast with about a sixth of its compute and a ninth of its parameters. The search also found something surprising. The efficient networks were thin and high in spatiotemporal resolution, the opposite of what image networks suggest.

Video Transformers choose which tokens each query sees

Now the input is tokens. A video Transformer reads $T = n_tn_s$ space-time tokens and must decide which tokens each query attends to. Recall how attention works (§4.5). Each token makes a query, a key, and a value. Its query is compared with the keys of the tokens it may look at, and its output is a weighted average of their values. The obvious choice is to let every query look at every token.

The attention core is the two products that involve every pair of tokens, the scores and the weighted sum of values. It costs $4T^2d$ FLOPs per layer (§4.7), and $T$ grows with the number of frames. A ViT-B/16, the base-size ViT with $16\times16$-pixel patches, reads a $224\times224$ image as 196 tokens. Sixteen frames multiply that by 8 with two-frame tubelets, or by 16 with one patch per frame. The core grows by the square, 64 or 256 times. The cheaper alternatives restrict each query to a structured subset of the keys.

That two-step pattern is the design of TimeSformer (Bertasius et al. 2021).

Three relatives complete the family.

• Axial attention goes one step further, with separate sublayers along time, width, and height.
• Sparse local-global attention first attends to nearby tokens, then to a strided subset of the whole clip.
• Space-only attention keeps just the spatial sublayer, so information never crosses frames until the final pooling. That is the image model on every frame from the opening.

What does the two-step structure cost in expressiveness? Simplify the block to see it. Ignore the residuals, and take the value and output maps as the identity, so that each sublayer only mixes tokens. Let $\mathbf{A}^{\text{time}}_{j}\in\R^{n_t\times n_t}$ be the temporal attention weights at spatial position $j$, and $\mathbf{A}^{\text{space}}_{i}\in\R^{n_s\times n_s}$ the spatial weights in time slice $i$. Write $\mathbf{x}_{i,j}$ for the input token at time $i$ and position $j$. Then the output at $(i, j)$ is

In plain words, the weight from input token $(i', j')$ to output token $(i, j)$ is a product of two attention weights, $[\mathbf{A}^{\text{space}}_i]_{jj'}[\mathbf{A}^{\text{time}}_{j'}]_{ii'}$. One is for the hop in time and one for the hop in space.

So factorizing is a structural prior, an assumption built into the wiring of the network, and not a free saving. Whether it helps depends on the data, as the results show. An interviewer who asks whether divided attention "loses information" wants this answer. Nothing is unreachable, but the patterns it can express are constrained.

The parameter gap is worth checking, because it says what a divided block contains. $121.4 - 85.9 = 35.5$ million parameters over 12 blocks is 2.96 million per block. That is $5d^2 = 5\cdot768^2 = 2.95$ million plus biases, so each block has five extra $d\times d$ matrices. Four are the temporal $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$. The fifth is a linear layer that the released code applies to the output of temporal attention. It initializes that layer to zero in every block but the first. At the start of training, then, the temporal sublayers of blocks 2 to 12 add nothing, and the network is close to the pretrained image ViT applied frame by frame. It is the same start-from-the-image-model idea as inflation.

The cell below runs a divided block in NumPy on the two-frame, four-token example and checks the reach directly. It perturbs one input token and asks which outputs change.

All 8 outputs change under the divided block, and only frame 0 changes under space-only attention. The divided block uses 48 score entries against joint attention's 64.

The widget shows the same reach on a larger grid. Pick a scheme, click a token to make it the query, and play the sublayers of one block to see which tokens have reached it so far. For the windows it plays two blocks, the second with its windows shifted by one token.

import numpy as np

def divided_attention(X, Wt, Ws):
    raise NotImplementedError

In PyTorch the same block is two reshapes around a fused attention kernel. The temporal sublayer is also exactly joint attention with a mask that allows only pairs at the same spatial position, and the cell checks that. The masked version computes all $T^2$ scores and throws most of them away. The reshaped version computes only $Tn_t$, which is the reason to reshape.

Four ways to factorize, compared

Arnab et al. (2021) mapped the design space with tubelet tokens in ViViT, comparing four ways to split attention.

What video attention costs, exactly

Every scheme above changes only which keys each query sees. That makes the cost easy to derive once for all of them. If a query sees $g$ keys, its attention core costs $4gd$ FLOPs, so the core of a sublayer costs $4Tgd$. Here is the tiny case at real scale. For ViT-B at 8 frames of 224 pixels, $T = 8\cdot196 = 1{,}568$. Joint attention has $g = 1{,}568$. Divided attention has $g = 8$ and then $g = 196$, a total of 204 keys per query, 7.7 times fewer.

In plain words, factorizing shrinks the attention core by the factor $T^2/(T(n_t + n_s)) = n_tn_s/(n_t + n_s)$. But it adds projections whose cost grows with $T$, not $T^2$. Compare whole layers. Divided attention is cheaper than joint attention if and only if

The right side comes from dividing both sides by $4Td$. At $d = 768$ and 8 frames of 224 pixels, the threshold is $1{,}536 + 8 + 196 = 1{,}740$ tokens. The clip has 1,568, so short, low-resolution clips sit just below the break-even point. The cell tabulates both layers at several clip sizes.

An interviewer who asks "is divided attention cheaper?" wants this answer. It is cheaper only above about $2d$ tokens, because on short clips the extra projections cost more than the attention they save.

The widget draws both layers' costs against the number of frames, on logarithmic axes, so the $T^2$ growth of joint attention shows as the steeper curve. Slide the frames to read off both costs, and switch to 448-pixel frames to see the crossing move left.

def video_attention_flops(n_t, n_s, d, scheme, n_h=None, n_w=None, L=1, L_t=0):
    raise NotImplementedError

def layer_flops(n_t, n_s, d, scheme, n_h=None, n_w=None):
    raise NotImplementedError

Reproducing published FLOPs, and the multiply-add trap

The formulas are exact, so they should reproduce the papers' numbers at real scale. They do, after one correction, which §27.2 explained and §29.2 already applied. Vision papers in this line, and the tools they use to count, report one multiply-add as one FLOP, half of the book's count. X3D says so explicitly. The cell converts our counts to multiply-adds and adds the cost of the patch or tubelet embedding, one matrix multiply per token.

Two details matter. First, TimeSformer's divided block has five extra $d\times d$ matrices, as the parameter count showed, so its layer is $34Td^2 + 4T(n_t + n_s)d$. Second, its headline "0.59 TFLOPs", in the comparison with SlowFast, is per video, 3 views of about 196 GFLOPs each. SlowFast's 1.97 TFLOPs in the same table is 30 views of 65.7 GFLOPs. Compare a per-view number from one paper with a per-video number from another and you are off by the number of views. Mix multiply-adds with FLOPs and you are off by another factor of 2.

Pyramids and windows bring locality back

A plain ViT keeps $T$ tokens of width $d$ in every layer. Convolutional networks instead shrink the resolution and widen the channels as depth grows. Early layers see fine detail cheaply, and late layers see the whole scene in a few rich features. Two Transformer families brought that structure back to video.

The first pools. MViT (Fan et al. 2021) uses pooling attention. Before attention, a strided pooling operator shrinks the queries, the keys, and the values over space. Pooling the queries shrinks the output, which is how each stage halves height and width. Pooling the keys and values shrinks what each query reads, so every query sees about the same number of keys in every stage. Over four stages MViT-B goes from 25,089 tokens of width 96 to 393 tokens of width 768.

The 9.9-point gap is the image story of §27.3 again. Without large pretraining, a model with the assumptions of a convolutional network built in, local detail first and a pyramid of resolutions, learns more from the same data than a plain ViT does.

The second family uses windows. Video Swin (Liu et al. 2022) extends the shifted windows of §27.5 to three dimensions. Attention runs inside non-overlapping windows of $8\times7\times7$ tokens (time, height, width). Alternate blocks shift the windows by half a window, so information crosses window borders. Each query sees a fixed $8\cdot7\cdot7 = 392$ keys, the same number as a query in MViT-B. The core is therefore $4\cdot392\cdot Td$, linear in $T$, which is the local attention of §12.6 along three axes. Swin-B pretrained on ImageNet-21k reaches 82.7 on Kinetics-400 at 282 GFLOPs times 12 views. Swin-L at 384 pixels reaches 84.9 with 50 views. Swin-B pretrained on Kinetics-400 reaches 69.6 on Something-Something v2.

Reading the big table

Here are the published numbers for the models in this section, with the conditions that make them comparable or not. GFLOPs are as reported (multiply-adds) per view, times the number of views.

The two-stream I3D accuracy is from its own paper, on the Kinetics test set of the time (later comparison tables list it at 75.7). Its cost and size are from the X3D paper, which gives no view count for it. The last row is the subject of §29.4 (Tong et al. 2022). It is a plain ViT with joint attention, and its pretraining, not its architecture, makes the difference.

So when a change improves Kinetics by half a point, do not conclude that the model uses motion. Test it on Something-Something v2, or on Diving-48, a benchmark of dives that differ only in their sequence of twists and somersaults. Run control experiments that break time while keeping appearance, by shuffling or reversing the frames of each test clip. A model that uses motion should fall sharply. A model that is a bag of frames will not notice.

The 2026 defaults, and why

Put the pieces together. Joint attention was the most accurate of ViViT's four models on Kinetics. In TimeSformer's study it came within 0.6 points of divided attention on Kinetics, and 1.0 on Something-Something, while divided attention had 35.5 million more parameters. It lost to factorizing for two reasons. It ran out of memory (TimeSformer's joint model did not fit at 448 pixels or 32 frames), and its FLOPs grow with $T^2$.

FlashAttention (Dao et al. 2022) removed the first reason. It computes attention in tiles and never stores the $T\times T$ score matrix, so attention memory grows only linearly in $T$ (§12.3). That leaves FLOPs, and the break-even analysis says factorizing saves little until $T$ is well above $2d$. What modern systems do next depends on what the encoder is for, and there are two answers.

Self-supervised video encoders are plain ViTs with joint space-time attention over tubelet tokens. They cut $T$ instead of restricting attention. Masked pretraining feeds the encoder only 10 percent of the tokens (§29.4). That is why VideoMAE could adopt a plain ViT with joint attention, and its authors name that attention's quadratic cost as the bottleneck that masking relieves. The V-JEPA models (Bardes et al. 2024) are plain ViTs on tubelet tokens as well. FlashAttention keeps their memory linear in $T$, and at a few thousand tokens per clip factorizing would save little.

Video language models, language models that read video tokens alongside text and answer in text (§29.5), factorize. Qwen2-VL (Wang et al. 2024) embeds each pair of frames as one time slice with $2\times14\times14$ tubelets (§29.2). Its vision encoder attends only within each slice, to the $n_s = n_hn_w$ tokens of one pair of frames. So no encoder layer mixes time beyond the two frames inside a tubelet. Qwen2.5-VL (Bai et al. 2025a) restricts attention further, to windows of $8\times8$ patches ($112\times112$ pixels) within a slice, in 28 of its 32 layers. The other four layers (7, 15, 23, and 31, counting from 0) attend over the whole slice. Neighboring $2\times2$ tokens are then merged into one. Time is mixed only inside the language model, whose attention runs over the merged tokens of all slices, each tagged with its position in space and time.

This is ViViT's factorized encoder at frontier scale. The language model plays the temporal Transformer, and it receives all of a slice's merged tokens instead of one vector per slice. Its cost follows from the theorem above. With $L_s$ encoder layers, the encoder's cores spend $L_s\cdot4Tn_sd$, which is $n_t$ times less than joint attention over the clip. The language model then pays for its own attention over the $T/4$ merged tokens.

The design reuses an image encoder unchanged and accepts any number of frames at any resolution. It puts temporal reasoning where the model's capacity and long context already are. Its weakness is that fine motion must survive per-slice encoding and merging before anything compares frames.

Long videos are handled on top of either design by sampling frames and budgeting tokens (§29.5). Sequence models whose cost is linear in length are an alternative for very long inputs, and with them recurrence returns. VideoMamba (Li et al. 2024) replaces attention with selective state-space layers, recurrences whose cost is linear in $T$. Video diffusion transformers, which generate video rather than classify it, face the same choice between joint and factorized attention over space-time latent patches (§34.5).

Read more

• Simonyan and Zisserman (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. The spatial and temporal streams and flow stacking, the paper that made deep video recognition work.
• Carreira and Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Inflated 3D ConvNets and Kinetics, with the clearest comparison of RGB, flow, and architectures of its time.
• Tran et al. (2018). A Closer Look at Spatiotemporal Convolutions for Action Recognition. R(2+1)D and the parameter-matched factorization, with careful ablations of mixed 2D and 3D designs.
• Feichtenhofer et al. (2019). SlowFast Networks for Video Recognition. Two frame rates, the channel ratio, and lateral connections, trained from scratch.
• Bertasius et al. (2021). Is Space-Time Attention All You Need for Video Understanding?. TimeSformer, the five attention schemes, and the Kinetics against Something-Something contrast.
• Arnab et al. (2021). ViViT: A Video Vision Transformer. Tubelets, the four factorized models with their cost and accuracy, and initialization from image models.
• Lin et al. (2019). TSM: Temporal Shift Module for Efficient Video Understanding. Zero-FLOP temporal modeling and its online variant.
• Lucas and Kanade (1981). An Iterative Image Registration Technique with an Application to Stereo Vision. The original least-squares registration behind Lucas-Kanade flow.

Watch next

▶

▶

▶