Chapter 29 · Section 29.2 · 70 min

Spatiotemporal tokens

This section is about how a video model turns a clip into tokens. A Transformer reads a sequence of vectors, called tokens, so the first thing every video Transformer does is cut the clip into pieces and turn each piece into one token. That step sets how many tokens the model pays for, what each token can see, and how the model learns where and when each piece came from. Get it wrong and the model is slow, blind to motion, or confused about time, and none of these mistakes raises an error.

Here is the problem in numbers. A Vision Transformer (ViT) cuts a $224\times224$ image into squares of $16\times16$ pixels, called patches, and makes one token from each, 196 tokens in all (§27.1). Hand it a 16-frame clip, and the obvious move is to cut every frame the same way. That gives $16\cdot196 = 3{,}136$ tokens. Attention compares every token with every other, so the cost of its scores grows with the square of the token count. For the clip that part costs $16^2 = 256$ times as much as for one image (§4.7). Worse, no token contains any motion. Each one is a still square from a single frame, so everything the model learns about movement must be assembled later, inside attention.

Most modern video encoders fix both problems the same way. They cut the clip into small blocks that span a few frames as well as a few pixels on a side, and make one token per block. These blocks are called tubelets. Each one looks like a brick cut from the stack of frames, so we will also call them bricks. With bricks two frames deep, the same clip becomes 1,568 tokens, and each token sees a little motion.

This section works the idea out exactly. We count the tokens, parameters, and FLOPs of a tubelet embedding. We show that it is a 3D convolution and also a single matrix multiply. We start it from a trained image model without losing what that model knows. Then we tell each token where and when it is, from learned tables to the rotary positions of today's video language models, and we follow how time in seconds reaches the language model. We end with a tokenizer that skips bricks that did not change.

The input is a clip of $T_f$ frames, already sampled from the video as in §29.1. We store it as $\mathbf{X}\in\R^{T_f\times H\times W\times C}$ , frames by height by width by channels, with $C = 3$ for color. PyTorch's 3D layers expect the layout $(C, T_f, H, W)$ instead, and the code always says which one it uses. The letter $T$ is kept for the number of tokens, as everywhere in the book.

Cutting a clip into bricks

Start from the image case. A ViT cuts an image of $H\times W$ pixels with $C$ channels into a grid of $P\times P$ patches. It flattens each patch into a vector of $P^2C$ numbers, row by row, with the channels last. It multiplies every patch vector by the same matrix $\mathbf{W}_E\in\R^{P^2C\times d}$ and adds the same bias $\mathbf{b}_E\in\R^d$ . Here $d$ is the model width, the length of every token vector. The result is one token of $d$ numbers per patch (§27.1).

A tubelet embedding does the same thing with one more axis. The brick is $P_t$ frames deep and $P\times P$ pixels across, so it holds $P_tP^2C$ numbers. We flatten it, multiply it by one shared matrix, and add one shared bias. The name tubelet comes from ViViT, one of the first video Transformers (Arnab et al. 2021). That paper writes the depth as $t$ , and we write $P_t$ because $t$ is a time index. Here is the cut on a clip small enough to check by hand.

✏️ Eight tokens from a 4 by 4 by 4 clip

Take a single-channel clip of 4 frames of $4\times4$ pixels. Let the pixel at frame $t$ , row $y$ , column $x$ hold the number $16t + 4y + x$ , so every pixel is labeled by its own coordinates. Cut it into bricks with $P_t = P = 2$ . There are 2 bricks along time, 2 along height, and 2 along width, so $2\cdot2\cdot2 = 8$ tokens of $2\cdot2\cdot2 = 8$ numbers each.

Token $(i, j, k) = (0, 1, 0)$ is the brick in time slot 0 (frames 0 and 1), row slot 1 (rows 2 and 3), and column slot 0 (columns 0 and 1). Frame 0 contributes pixels 8, 9, 12, 13, and frame 1 contributes 24, 25, 28, 29. Flattened time first, then row, then column, the token is $(8, 9, 12, 13, 24, 25, 28, 29)$ .

The tokens are listed in time-row-column order. With $n_h = n_w = 2$ slots along height and width, token $(i, j, k)$ is row $r = (i\,n_h + j)\,n_w + k$ of the token matrix. For $(0, 1, 0)$ that is $r = 2$ . The first four rows come from frames 0 and 1, the last four from frames 2 and 3.

The cell reproduces the example with a reshape and a transpose. Then it shows the most common bug in video tokenization, which is to reshape the clip straight into rows and skip the transpose. The rows have the right shape, but they are not bricks. Row 2 becomes pixels 16 to 23, the first two rows of frame 1.

Python · cell 1

The first reshape splits each axis into a slot and an offset inside the brick, giving the shape $(n_t, P_t, n_h, P, n_w, P)$ . The transpose moves the three slot axes to the front, so each brick's numbers sit together. The final reshape flattens each brick into one row. Here is the general statement.

📐 Tubelet embedding

Let $\mathbf{X}\in\R^{T_f\times H\times W\times C}$ be a clip, $P_t$ the temporal and $P$ the spatial tubelet size, and $d$ the model width. The token grid has

n_t = \Big\lfloor\frac{T_f}{P_t}\Big\rfloor,\qquad n_h = \Big\lfloor\frac{H}{P}\Big\rfloor,\qquad n_w = \Big\lfloor\frac{W}{P}\Big\rfloor

slots along time, height, and width, $n_s = n_hn_w$ tokens per time slot, and $T = n_tn_s = n_tn_hn_w$ tokens in all. Frames and pixels beyond $n_tP_t$ , $n_hP$ , and $n_wP$ are dropped. For $0\le i<n_t$ , $0\le j<n_h$ , $0\le k<n_w$ , token $(i, j, k)$ is

\mathbf{z}_{ijk} = \operatorname{vec}\big(\mathbf{X}[iP_t{:}(i+1)P_t,\ jP{:}(j+1)P,\ kP{:}(k+1)P,\ :]\big)^\top\mathbf{W}_{\text{tub}} + \mathbf{b}^\top\in\R^{1\times d}.

Here $\operatorname{vec}$ flattens the brick time first, then rows, then columns, with the channels last, so pixel $(\tau, u, v)$ of the brick in channel $c$ lands at index $((\tau P + u)P + v)C + c$ . The weights $\mathbf{W}_{\text{tub}}\in\R^{(P_tP^2C)\times d}$ (equivalently a tensor of shape $P_t\times P\times P\times C\times d$ ) and the bias $\mathbf{b}\in\R^d$ are shared by every brick. The tokens are ordered by time slot, then row, then column, so $(i, j, k)$ is row $(i\,n_h + j)\,n_w + k$ of the token matrix $\mathbf{Z}\in\R^{T\times d}$ .

In plain words, cut the clip into bricks, flatten each brick, multiply every brick by the same matrix, add the same bias, and list the results time slot by time slot. With $P_t = 1$ this is the image patch embedding of §27.1 applied to every frame on its own.

Leftovers are dropped. ViViT writes the floors explicitly, and PyTorch's nn.Conv3d without padding does the same. Clips are usually sampled so that nothing is left over. But a 17-frame clip with $P_t = 2$ silently loses its last frame, and a $230\times230$ frame with $P = 16$ silently loses a 6-pixel strip along its bottom and right edges.

The most common setting is 16 frames at $224\times224$ with $2\times16\times16$ tubelets. Two video encoders trained by masking use it, VideoMAE (Tong et al. 2022) and the spatiotemporal masked autoencoder of Feichtenhofer et al. (2022), often called MAE-ST. Both hide most of the tubelets and train the model to reconstruct them (§29.4).

What a token costs, and what it sees

The embedding has $P_tP^2Cd$ weights plus $d$ biases. ViT-B, the base-size Vision Transformer, has 12 layers of width $d = 768$ . With $2\times16\times16$ tubelets its embedding has $2\cdot16\cdot16\cdot3\cdot768 = 1{,}179{,}648$ weights and 768 biases, twice the image model's patch embedding. Qwen2-VL, an open vision language model that reads images and video (Wang et al. 2024), uses $2\times14\times14$ tubelets at width 1,280 with no bias. That is $2\cdot14\cdot14\cdot3\cdot1280 = 1{,}505{,}280$ weights.

The embedding's compute is one matrix multiply, a $T\times P_tP^2C$ matrix times a $P_tP^2C\times d$ one. Multiplying an $m\times k$ matrix by a $k\times n$ one takes $mkn$ multiply-adds, and we count each multiply-add as 2 FLOPs, as in §4.7. Try it on the $4\times4\times4$ clip of the example with $d = 1$ . Cut into $1\times2\times2$ patches, it gives 16 tokens of 4 numbers, so 64 multiply-adds. Cut into $2\times2\times2$ bricks, it gives 8 tokens of 8 numbers, again 64. That is no accident.

In plain words, every pixel value is multiplied by $d$ weights exactly once. For the 16-frame clip and ViT-B that is $2\cdot16\cdot224\cdot224\cdot3\cdot768 = 3.70$ GFLOPs, about 1 percent of the encoder that follows. We check this at real scale near the end of the section.

So the brick shape leaves the embedding's cost alone, but it changes everything after it. Each Transformer layer spends about $24Td^2$ FLOPs on its four $d\times d$ projections and its MLP of width $4d$ . It spends another $4T^2d$ on the attention core, the scores and the weighted sum of values (§4.7). Doubling $P_t$ halves $T$ . That halves the first term and quarters the second. For ViT-B on 16 frames at 224, the whole 12-layer encoder costs about 899 GFLOPs with per-frame tokens and 361 GFLOPs with $P_t = 2$ , a factor of 2.5.

What you give up is time resolution inside the network. With $P_t = 2$ the encoder never has separate tokens for frames $2i$ and $2i + 1$ , because one linear map combines both into a single token. That map can represent a little motion, since its weights can subtract one frame from the other. But in ViT-B it maps $1{,}536$ pixel values to $d = 768$ numbers, so it cannot keep everything about the pair. That is why $P_t$ stays small, almost always 2.

The choice also interacts with sampling. Take a training clip of 16 frames, read every 4th frame of a video at 30 frames per second. It spans about 2 seconds, and one time slot covers 8 source frames, about a quarter of a second. An interviewer who asks you to size a video model expects exactly this chain, from frames and tubelet shape to tokens, to the two cost terms, to how much time one token spans.

Warm-up

Count the tokens

Write three counting functions for the tubelet embedding of the definition.

tubelet_tokens(T_f, H, W, P_t, P) returns the tuple (n_t, n_h, n_w, T) of Python ints. Leftover frames and pixels are dropped.
tubelet_params(P_t, P, C, d, bias=True) returns the number of parameters, the weights plus the bias if there is one.
tubelet_flops(T_f, H, W, P_t, P, C, d) returns the forward FLOPs of the embedding's matrix multiply. Count a multiply-add as 2 FLOPs and ignore the bias.

def tubelet_tokens(T_f, H, W, P_t, P):
    n = (T_f * H * W) // (P * P)      # placeholder: forgets P_t and the floors per axis
    return (0, 0, 0, n)

def tubelet_params(P_t, P, C, d, bias=True):
    raise NotImplementedError

def tubelet_flops(T_f, H, W, P_t, P, C, d):

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The widget draws a clip as a block, with frame 0 in front and time running back. Pick a tubelet depth and watch the grid of bricks, the token count, and the attention cost relative to one image change.

A tubelet embedding is a 3D convolution

A convolution slides a small filter over its input and takes a dot product at every position. §0.8 wrote the 2D case with $F$ filters, a $k\times k$ kernel, stride $s$ , and no padding as $Y_{f,i,j} = b_f + \sum_c\sum_u\sum_v W_{f,c,u,v}\,X_{c,\,si+u,\,sj+v}$ . Along an axis of length $n$ its output has $\lfloor(n - k)/s\rfloor + 1$ positions. A 3D convolution adds a kernel axis along time. Take kernel $(k_t, k, k)$ , strides $(s_t, s, s)$ , PyTorch's layout $\mathbf{X}\in\R^{C\times T_f\times H\times W}$ , and filters $\mathbf{W}_{\text{conv}}\in\R^{F\times C\times k_t\times k\times k}$ . Then

Y_{f,i,j,l} = b_f + \sum_{c=0}^{C-1}\sum_{\tau=0}^{k_t-1}\sum_{u=0}^{k-1}\sum_{v=0}^{k-1} W_{f,c,\tau,u,v}\;X_{c,\;s_ti+\tau,\;sj+u,\;sl+v}.

The output has $\lfloor(T_f - k_t)/s_t\rfloor + 1$ positions along time, and $\lfloor(H - k)/s\rfloor + 1$ and $\lfloor(W - k)/s\rfloor + 1$ along height and width. We write $l$ for the output column here because $k$ is the kernel size.

Now set the stride equal to the kernel. The windows then sit side by side with no overlap and no gaps, which is exactly how the bricks tile the clip.

∂ Proof that the convolution and the tubelet embedding agree

The output size along time is $\lfloor(T_f - P_t)/P_t\rfloor + 1 = \lfloor T_f/P_t - 1\rfloor + 1 = \lfloor T_f/P_t\rfloor = n_t$ , since subtracting the integer 1 commutes with the floor. The same holds along height and width.

Fix an output position $(i, j, k)$ and a filter $f$ . With $s_t = P_t$ and $s = P$ , the input index $(P_ti + \tau,\ Pj + u,\ Pk + v)$ for $0\le\tau<P_t$ and $0\le u, v<P$ runs exactly over the frames, rows, and columns of brick $(i, j, k)$ . By the definition of $\operatorname{vec}$ , entry $((\tau P + u)P + v)C + c$ of the flattened brick is that pixel in channel $c$ . The map $(\tau, u, v, c)\mapsto((\tau P + u)P + v)C + c$ is a bijection onto $\{0, \dots, P_tP^2C - 1\}$ , because it writes a number in mixed radix with digits $\tau, u, v, c$ . So the convolution sum is the dot product of the flattened brick with column $f$ of $\mathbf{W}_{\text{tub}}$ , term by term. Adding $b_f$ to both gives $\mathbf{z}_{ijk}$ .

Finally, every frame index $t<n_tP_t$ is written uniquely as $t = P_ti + \tau$ with $0\le\tau<P_t$ , by division with remainder, and likewise for rows and columns. So each used pixel lies in exactly one brick. $\blacksquare$

In plain words, a convolution is a small dense layer applied to every window. When the stride equals the kernel along all three axes, the windows tile the clip, so a dense layer on every window is exactly a dense layer on every brick. This is the patch result of §27.1 with one more axis.

It also explains why the matrix-multiply form is cheap to build. The usual way to turn a convolution into one matrix multiply, called im2col, copies every window into a row of a matrix (§0.8). Overlapping windows make it copy each pixel many times. Here the windows are disjoint, so building that matrix copies nothing. It is a reshape and a transpose.

Here is the recipe in NumPy with the channels last.

Crop the clip of shape $(T_f, H, W, C)$ to $(n_tP_t, n_hP, n_wP, C)$ .
Reshape it to $(n_t, P_t, n_h, P, n_w, P, C)$ .
Transpose it to $(n_t, n_h, n_w, P_t, P, P, C)$ .
Reshape it to $(T, P_tP^2C)$ and multiply by $\mathbf{W}_{\text{tub}}$ reshaped to $(P_tP^2C, d)$ .

The transpose does two jobs, and each can go wrong silently. First, it puts the slot axes in front, in time-row-column order. Get that wrong and a token mixes pixels from different bricks. Second, it keeps each brick's own axes in the order the weight uses, which is the order of $\operatorname{vec}$ . Get that wrong and a pretrained embedding is applied to permuted pixels. Neither mistake raises an error.

The cell checks the theorem on a clip whose last frame and last 4 columns do not fit. It computes the convolution sum one output position at a time and compares it with the reshape and one matrix multiply.

Python · cell 2

There are 64 tokens ( $4\cdot4\cdot4$ ), and the two computations agree exactly. Frame 8 and columns 32 to 35 never entered a token. Writing this without loops is the video version of the patchify question that labs ask in interviews, and the next exercise asks you to do it.

Interview

Tubelet embedding without loops

Write tubelet_embed(X, W, b, P_t, P). The clip X has shape (T_f, H, W_in, C), the weights W have shape (P_t, P, P, C, d), and the bias b has shape (d,). Return the tokens as an array of shape (n_t * n_h * n_w, d), in time-row-column order, with leftover frames and pixels dropped. Use only slicing, reshape, transpose, and one matrix multiply, no loops. The tests catch the usual mistakes, which are a missing crop, a missing transpose, the wrong flattening order, and leftovers leaking into tokens.

import numpy as np

def tubelet_embed(X, W, b, P_t, P):
    T_f, H, W_in, C = X.shape
    d = W.shape[-1]
    n_t, n_h, n_w = T_f // P_t, H // P, W_in // P
    rows = X.reshape(n_t * n_h * n_w, -1)            # bug: no crop, no transpose, so rows are not bricks
    return rows @ W.reshape(-1, d) + b

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Starting from an image model

Labeled video is scarce next to labeled images. Kinetics-400, the standard action dataset of short YouTube clips in 400 classes (§29.1), has about 240,000 training clips. Image models are trained on hundreds of millions of images or more. So a ViT trained from scratch on video does poorly, and VideoMAE measured how poorly.

Top-1 accuracy is the fraction of test clips whose highest-scoring class is the correct one. Something-Something v2 is a dataset of people moving objects, where the label often depends on the direction of motion, so a single frame cannot solve it. Its training set has about 169,000 clips. The gap of 29 points shows how far that is from enough data to train a ViT from nothing.

So we want to start a video model from an image model. Every Transformer weight carries over unchanged. The only new weights are the extra temporal taps of the tubelet embedding and the positions for time. A tap is one frame's slice of the filter, $\mathbf{W}_{\text{tub}}[\tau]$ for $\tau = 0, \dots, P_t - 1$ . How should the taps be set?

The guiding idea comes from Carreira and Zisserman (2017), whose Inflated 3D ConvNet, I3D, turned whole image CNNs into video CNNs (§29.3). Turn an image into a boring video by copying it into every frame. A good initialization should make the video model's response to the boring video equal the image model's response to the image. They called this the boring-video fixed point. Here are two ways to get it, on numbers small enough to do in your head.

✏️ Two ways to start a two-frame filter, by hand

Shrink everything to one pixel, one channel, and width $d = 1$ . The image model's patch embedding is then the single number $w = 6$ , so an image pixel of value 2 gives the token $6\cdot2 = 12$ . The video filter has $P_t = 2$ taps, $w_0$ for frame 0 and $w_1$ for frame 1.

Copy and divide. Set $w_0 = w_1 = 6/2 = 3$ . On the boring video $(2, 2)$ the token is $3\cdot2 + 3\cdot2 = 12$ , the image token. On a moving video $(2, 4)$ it is $3\cdot2 + 3\cdot4 = 18 = 6\cdot3$ , the image token of the average frame, 3.

Center only. Set $w_0 = 0$ and $w_1 = 6$ . On the boring video the token is $0\cdot2 + 6\cdot2 = 12$ , again the image token. On the moving video it is $6\cdot4 = 24$ , the image token of frame 1 alone.

Without the division by 2, copying would give $6\cdot2 + 6\cdot2 = 24$ on the boring video, twice what the image model ever produced.

In the general case the filter has $P_t$ temporal taps instead of two, and each tap is a whole patch embedding of shape $P\times P\times C\times d$ rather than one number.

📐 Inflation and central-frame initialization

Let $\mathbf{W}_{2D}\in\R^{P\times P\times C\times d}$ be an image model's patch embedding, the matrix $\mathbf{W}_E\in\R^{P^2C\times d}$ above with its rows indexed by pixel row, pixel column, and channel. Let $\mathbf{b} = \mathbf{b}_E$ be its bias. Index the temporal taps of $\mathbf{W}_{\text{tub}}\in\R^{P_t\times P\times P\times C\times d}$ by $\tau = 0, \dots, P_t - 1$ . Both methods keep $\mathbf{b}$ unchanged.

Inflation (Carreira and Zisserman 2017, called filter inflation in ViViT) copies the image filter to every tap and divides by the depth, $\mathbf{W}_{\text{tub}}[\tau] = \mathbf{W}_{2D}/P_t$ for every $\tau$ .
Central-frame initialization (Arnab et al. 2021) puts the image filter at the central tap and zeros elsewhere, $\mathbf{W}_{\text{tub}}[\lfloor P_t/2\rfloor] = \mathbf{W}_{2D}$ and $\mathbf{W}_{\text{tub}}[\tau] = \mathbf{0}$ for $\tau\ne\lfloor P_t/2\rfloor$ .

Inflation spreads the image filter evenly over the taps, and central-frame initialization puts all of it on one tap. The hand example showed that both pass the boring-video test. The theorem below says the same holds for any depth $P_t$ and any image filter.

∴ The boring-video fixed point

Let the clip be boring, every frame equal to the same image, $\mathbf{X}[t] = \mathbf{X}_{\text{img}}$ for all $t$ . Under either initialization, every token equals the image model's patch token at the same spatial position, $\mathbf{z}_{ijk} = \mathbf{e}_{(j,k)}$ for every time slot $i$ , where $\mathbf{e}_{(j,k)} = \operatorname{vec}(\text{patch }(j,k)\text{ of }\mathbf{X}_{\text{img}})^\top\mathbf{W}_{2D} + \mathbf{b}^\top$ .

Proof. The flattened brick is the patches of its $P_t$ frames laid end to end, so the token is a sum over taps, $\mathbf{z}_{ijk} = \sum_{\tau}\mathbf{x}_{\tau}^\top\mathbf{W}_{\text{tub}}[\tau] + \mathbf{b}^\top$ . Here $\mathbf{x}_\tau$ is patch $(j, k)$ of frame $iP_t + \tau$ , flattened, and $\mathbf{W}_{\text{tub}}[\tau]$ is read as a $P^2C\times d$ matrix. On a boring video every $\mathbf{x}_\tau$ equals the image patch $\mathbf{x}$ . Under inflation the sum is $P_t$ equal terms $\mathbf{x}^\top\mathbf{W}_{2D}/P_t$ , which add to $\mathbf{x}^\top\mathbf{W}_{2D}$ . Under central-frame initialization only the term $\tau = \lfloor P_t/2\rfloor$ is nonzero, and it equals $\mathbf{x}^\top\mathbf{W}_{2D}$ . In both cases adding $\mathbf{b}^\top$ gives $\mathbf{e}_{(j,k)}$ . $\blacksquare$

The same sum says what each method does on a video that is not boring, as the hand example showed. The map is linear, so the inflated token is the image embedding of the brick's frames averaged, $\big(\tfrac{1}{P_t}\sum_\tau\mathbf{x}_\tau\big)^\top\mathbf{W}_{2D} + \mathbf{b}^\top$ . At initialization the model sees motion-blurred frames. The central-frame token is the image embedding of one frame of each brick, frame $iP_t + \lfloor P_t/2\rfloor$ , and the other frames are ignored. ViViT's simplest tokenizer, uniform frame sampling, does exactly this on purpose. It keeps $n_t$ evenly spaced frames, here one in every $P_t$ , and cuts each kept frame into image patches. So at initialization a central-frame model is the uniform-sampling model, which is what ViViT means when it says the filter "behaves like uniform frame sampling". Training then teaches the other taps to use time.

The cell builds both initializations from a random image filter. It checks the fixed point on a static clip. Then it embeds a clip whose content moves 3 pixels per frame and confirms the two readings above.

Python · cell 3

On the static clip both initializations reproduce the image tokens to rounding error. On the moving clip they part ways, and the last two lines confirm that inflation embeds the average frame and central-frame initialization embeds frame 1.

The fixed point is about the first layer, but more is true. Suppose every token attends to every other (joint space-time attention), the positions added to the tokens do not depend on the time slot, and the model averages its output tokens at the end. Then the whole video ViT reproduces the image ViT on a boring video, exactly. The reason is one small fact about attention. If every key and value appears $n_t$ times, the attention output does not change, because the factor $n_t$ cancels in the softmax. The box below proves it and shows what breaks the equality.

🔬 The whole network reproduces the image model

For a CNN, the I3D paper's argument settles this. Convolving a signal that is constant in time gives a signal that is constant in time, and pointwise operations and pooling keep it so. (This is exact away from the ends of the clip, where temporal padding intervenes.) A ViT needs two steps.

Duplicated keys leave attention unchanged. Let a query $\mathbf{q}$ attend to keys $\mathbf{k}_1, \dots, \mathbf{k}_n$ with values $\mathbf{v}_1, \dots, \mathbf{v}_n$ , and let a second sequence hold each pair $(\mathbf{k}_j, \mathbf{v}_j)$ exactly $m$ times. Write $s_j = \mathbf{q}^\top\mathbf{k}_j/\sqrt{d_k}$ for the scaled score of key $j$ , with $d_k$ the key width. On the second sequence the output is $\sum_j m\,e^{s_j}\mathbf{v}_j\big/\sum_j m\,e^{s_j}$ . The factor $m$ cancels, leaving the first sequence's output.

The whole network, by induction over the blocks. Take a ViT with joint space-time attention, in which every token attends to every other. Let the position vector added to token $(i, j, k)$ depend only on $(j, k)$ . Run it on a boring video with either initialization. By the fixed point, the $n_t$ time slices of its input are identical copies of the image model's tokens. Suppose this holds at the input of some block. LayerNorm, the projections, the MLP, and the residual additions act on each token separately, so they keep the property. In joint attention a query from any slice sees every image key and value exactly $n_t$ times. By the first step, its output is the image model's output for that query. So the property holds at the block's output, and by induction at the network's output. The mean over all $n_tn_s$ tokens is then the mean over the $n_s$ image tokens. $\blacksquare$

The condition on positions is real. ViViT meets it by building its space-time position table from the image table repeated along time. A table with separate time and space parts (next part) meets it if its time vectors start equal. A class token, one extra learned token whose output is read as the summary, breaks the exact equality. It appears once, while each patch key appears $n_t$ times, so the patches get more weight than in the image model. The cell checks both claims on a random three-block Transformer.

Python · cell 4

So, at initialization, a video ViT built this way gives the image ViT's output on every boring video, up to rounding. Which of the two initializations trains better is an empirical question, and ViViT answered it.

A randomly initialized embedding throws away the image model's first layer and lands 6 points behind central frame. Central frame beats inflation by 1.6 points and uniform frame sampling by 0.7. A plausible reading, which the paper does not test, is that central frame starts from a model that is already good (the uniform-sampling model), while inflation starts from one that sees blurred inputs.

Later work kept the conclusion that tubelets should start from the image filter. VideoMAE, for its baseline that starts from a ViT-B pretrained as an image masked autoencoder on ImageNet-1k, inflates the image patch embedding into its tubelet embedding. Qwen2-VL feeds each image as two identical frames, a boring two-frame video. Its $2\times14\times14$ embedding therefore acts on an image as a 2D convolution whose filter is the sum of its two taps, $\mathbf{W}_{\text{tub}}[0] + \mathbf{W}_{\text{tub}}[1]$ . In an interview, "how would you start a video model from an image model?" is answered by the fixed point, the two recipes, and these numbers.

Core

Inflate an image model

Write inflate_patch_embedding(W2d, P_t, mode). It turns an image patch embedding W2d of shape (P, P, C, d) into a tubelet embedding of shape (P_t, P, P, C, d).

With mode="inflate", copy the filter into every temporal tap and divide by P_t.
With mode="central", put the filter at tap P_t // 2 and zeros at every other tap.

Do not modify W2d. The tests check the boring-video fixed point with their own embedding code.

import numpy as np

def inflate_patch_embedding(W2d, P_t, mode):
    return np.repeat(W2d[None], P_t, axis=0)          # placeholder: copies without rescaling, ignores mode

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Telling each token where and when it is

Self-attention cannot see order. A Transformer without positions treats its input as a set. Shuffle the tokens, and the outputs are shuffled the same way and otherwise unchanged (§4.6). A video model without positions reads a bag of bricks. It cannot tell which brick came first, so it cannot tell a door opening from a door closing. Each token needs its time slot $i$ , row $j$ , and column $k$ .

§27.1 did this for the two spatial axes of an image. Video adds a third axis, and with it two new problems. The tables get large, and one step in time means different durations at different frame rates. We go from the simplest scheme to the one frontier models use.

Learned tables, joint and separable

An image ViT marks position by adding a learned vector, a position embedding, to each patch token, one vector per grid position (§27.1). The direct extension to video gives every one of the $T = n_tn_hn_w$ grid positions of the clip its own learned vector. That is $n_tn_hn_wd$ parameters. For 1,568 tokens at $d = 768$ it is 1,204,224 parameters, about as many as the tubelet embedding itself. And a model trained on 8 time slots has no vector at all for slot 9. ViViT uses such a joint table and initializes it by repeating the image model's spatial table along time, which meets the condition of the whole-network result above.

The cheaper option learns one table for time and one for space, and adds them.

MAE-ST uses separable embeddings in its encoder, noting that the "separable implementation prevents the size of positional embeddings growing too large in 3D". It reports that a fixed sine-cosine variant works similarly. At $n_t = 8$ , $n_s = 196$ , $d = 768$ , the separable tables have $(8 + 196)\cdot768 = 156{,}672$ parameters, 7.7 times fewer than the joint one.

The price is structure. The position part of every token is a time vector plus a space vector. Any interaction between when and where must be computed by the network rather than stored in the table, which for most data is no loss. The gain is that each table can be resized on its own. To read 32 time slots with a model trained on 8, interpolate the 8 temporal vectors to 32 along one axis and fine-tune briefly. That is the one-dimensional version of the 2D resizing in §27.1.

Separable tables also make initialization easy. Set $\mathbf{p}^{\text{space}}$ to the image model's table and every $\mathbf{p}^{\text{time}}_i$ to the same vector (zero, for instance), and the positions no longer depend on the time slot, as the whole-network result needs.

Three-dimensional sines and cosines

The fixed alternative extends the two-dimensional sine-cosine table of §27.1 to three axes. Recall the one-dimensional encoding of §5.4. At width $m$ , position $x$ maps to the vector $\mathbf{s}_m(x)$ with entries $\sin(x\omega_l)$ and $\cos(x\omega_l)$ in columns $2l$ and $2l + 1$ , where $\omega_l = b^{-2l/m}$ . The scalar base $b$ is 10,000 there, and it is unrelated to the bias $\mathbf{b}$ above. The useful identity is $\mathbf{s}_m(x)^\top\mathbf{s}_m(x') = \sum_l\cos\big((x - x')\omega_l\big)$ , a function of the offset $x - x'$ alone. For video, give each axis its own block of the vector.

In plain words, the similarity of two positions is a time term plus a row term plus a column term, and each depends only on how far apart the two tokens are along that axis. The cell builds the table for VideoMAE's $8\times14\times14$ grid with $d = 768$ split into three parts of 256. It checks the identity by shifting two tokens together, and it compares the parameter counts of the two learned tables.

Python · cell 5

VideoMAE takes a cruder route that is worth knowing because it works. Its encoder adds a fixed one-dimensional sinusoidal table over the 1,568 flattened positions. A token's position is then its raster index $(i\,n_h + j)\,n_w + k$ , its row in the token matrix. The network must learn on its own that index $r$ and index $r + 196$ are the same place one time slot apart.

Three-dimensional RoPE

The schemes so far add a vector to each token. Attention scores then depend on absolute positions, through the cross terms between content and position (§5.4 showed this in one dimension). Video language models instead use rotary position embedding, RoPE, which rotates queries and keys and adds nothing.

Here is RoPE in plain words, from §5.5. Split a query or key $\mathbf{q}\in\R^{d_k}$ into $d_k/2$ pairs of coordinates, pair $j$ being coordinates $(2j, 2j+1)$ . Treat each pair as the hand of a clock that turns at its own speed, $\omega_j = b^{-2j/d_k}$ radians per position. For the token at position $m$ , turn pair $j$ by the angle $m\omega_j$ . The score of a rotated query and a rotated key then depends only on their content and the offset between them, $(\mathbf{R}_m\mathbf{q})^\top(\mathbf{R}_n\mathbf{k}) = \mathbf{q}^\top\mathbf{R}_{n-m}\mathbf{k}$ .

For video, give each token a position triple (time, row, column), and let each clock be driven by one of the three coordinates. Here is the idea on numbers before the definition.

✏️ A 3D RoPE score by hand, and the same score after a shift

Use one pair per axis, so $d_k = 6$ , with toy speeds chosen for easy arithmetic, $\omega = 0.5$ for time, $0.3$ for height, and $0.2$ for width. Take the query and key to be $(1, 0)$ in every pair. A pair rotated by $\alpha$ is $(\cos\alpha, \sin\alpha)$ , and the dot product of two such pairs is $\cos(\alpha - \beta)$ .

Let the query sit at $\mathbf{m} = (1, 0, 2)$ and the key at $\mathbf{n} = (3, 1, 2)$ . The offset $\mathbf{n} - \mathbf{m}$ is 2 in time, 1 in height, and 0 in width, so the score is

\cos(2\cdot0.5) + \cos(1\cdot0.3) + \cos(0) = 0.5403 + 0.9553 + 1 = 2.4956 .

Shift both tokens by $(5, 2, 1)$ , to $(6, 2, 3)$ and $(8, 3, 3)$ . Every offset is unchanged, so every cosine is unchanged, and the score is again 2.4956.

📐 3D RoPE (multimodal RoPE, M-RoPE)

Give each token a position triple $\mathbf{m} = (m_t, m_h, m_w)$ of integers (time, height, width). Partition the pair indices $\{0, \dots, d_k/2 - 1\}$ into three sets $J_t$ , $J_h$ , $J_w$ of sizes $d_t/2$ , $d_h/2$ , $d_w/2$ , with $d_t + d_h + d_w = d_k$ . The rotation $\mathbf{R}_{\mathbf{m}}\in\R^{d_k\times d_k}$ is block diagonal, and its block for pair $j\in J_a$ is the $2\times2$ rotation $\mathbf{R}(m_a\omega_j)$ , with the same frequencies $\omega_j = b^{-2j/d_k}$ as 1D RoPE. Queries and keys are rotated after their projections, $\mathbf{q}\mapsto\mathbf{R}_{\mathbf{m}}\mathbf{q}$ and $\mathbf{k}\mapsto\mathbf{R}_{\mathbf{n}}\mathbf{k}$ . Values are not rotated. Two partitions are in use.

Sectioned (Wang et al. 2024, Qwen2-VL). Contiguous blocks, $J_t = \{0, \dots, d_t/2 - 1\}$ , then $J_h$ , then $J_w$ . Qwen2-VL uses $(d_t, d_h, d_w)/2 = (16, 24, 24)$ pairs of a 128-wide head.
Interleaved (Bai et al. 2025b, Qwen3-VL). The axes take turns. Pair $j$ goes to height if $j\equiv1\pmod 3$ and $j<3\cdot d_h/2$ , to width if $j\equiv2\pmod 3$ and $j<3\cdot d_w/2$ , and to time otherwise. Qwen3-VL uses $(24, 20, 20)$ pairs.

With 6 pairs split two per axis, the sectioned layout assigns pairs 0 to 5 to time, time, height, height, width, width. The interleaved layout assigns them to time, height, width, time, height, width.

∴ The relative property in three dimensions

For any query $\mathbf{q}$ , key $\mathbf{k}$ , and position triples $\mathbf{m}$ , $\mathbf{n}$ ,

(\mathbf{R}_{\mathbf{m}}\mathbf{q})^\top(\mathbf{R}_{\mathbf{n}}\mathbf{k}) = \sum_{a\in\{t,h,w\}}\ \sum_{j\in J_a}\mathbf{q}_{(j)}^\top\,\mathbf{R}\big((n_a - m_a)\,\omega_j\big)\,\mathbf{k}_{(j)},

where $\mathbf{q}_{(j)}, \mathbf{k}_{(j)}\in\R^2$ are pair $j$ of the query and key. The score depends on positions only through the offset $\mathbf{n} - \mathbf{m}$ . It is a sum of a time term, a height term, and a width term, each depending on one coordinate of the offset. If all three coordinates of each token are equal ( $m_t = m_h = m_w = m$ , and likewise for $\mathbf{n}$ ), the rotation is exactly 1D RoPE at positions $m$ and $n$ , for any partition. The proof is in the derivation below.

∂ Proof of the 3D relative property

Both rotations are block diagonal with the same block structure, so the dot product is a sum over pairs, $\sum_j(\mathbf{R}(m_{a(j)}\omega_j)\mathbf{q}_{(j)})^\top(\mathbf{R}(n_{a(j)}\omega_j)\mathbf{k}_{(j)})$ , where $a(j)$ is the axis that owns pair $j$ . For a $2\times2$ rotation, $\mathbf{R}(\alpha)^\top = \mathbf{R}(-\alpha)$ and $\mathbf{R}(-\alpha)\mathbf{R}(\beta) = \mathbf{R}(\beta - \alpha)$ , the two facts §5.5 used. So each term is $\mathbf{q}_{(j)}^\top\mathbf{R}\big((n_{a(j)} - m_{a(j)})\omega_j\big)\mathbf{k}_{(j)}$ , and grouping the pairs by axis gives the formula. If every coordinate of $\mathbf{m}$ equals $m$ , every pair is rotated by $m\omega_j$ whichever axis owns it, which is the 1D rotation $\mathbf{R}_m$ . The same holds for $\mathbf{n}$ . $\blacksquare$

In plain words, each frequency pair is a clock that one axis drives. The score between two tokens depends on how far apart they are in time, in rows, and in columns, never on where they are. Text, whose three coordinates are always equal, sees ordinary RoPE, so the language model keeps everything it learned on text.

This differs from the axial 2D RoPE of §27.1, which gives each axis its own complete ladder of frequencies, from fast to slow. M-RoPE deals a single ladder out among three axes. So which pairs an axis receives decides which distances it can resolve, and the next part shows that this matters.

The cell implements 3D RoPE for both partitions. It checks the shift invariance with random queries and keys at the example's positions, and it checks that a text token gets plain 1D RoPE.

Python · cell 6

Which frequencies each axis gets

The partition looks like bookkeeping, but it decides what each axis can see. The frequencies fall geometrically with $j$ , so sectioning gives time the fastest clocks and width the slowest. The cell reads off both released configurations, with head width 128 and base $10^6$ for Qwen2-VL and $5\cdot10^6$ for Qwen3-VL (from their Hugging Face configuration files). For each axis it prints the fastest and slowest frequency and the largest angle an offset of 30 positions can produce.

Python · cell 7

Two problems show up in Qwen2-VL's sectioned layout. First, width gets only frequencies below $1.8\cdot10^{-4}$ radians per step. A column offset of 30 turns its pairs by at most 0.005 radians, a third of a degree, so the language model's attention can barely tell columns apart through RoPE. (The vision encoder has its own 2D RoPE, so spatial layout also reaches the language model through the content of the visual tokens.)

Second, time gets only fast clocks. The slowest turns once every $2\pi/0.039\approx160$ positions. Beyond about 160 time steps every time clock has wrapped around at least once, so a distant frame can land in nearly the same phases as a near one. Wei et al. (2025) document exactly this. They place distractor images, similar to a target frame, around it in a long video, and show that a sectioned M-RoPE is misled by them. They propose giving time the low frequencies instead. Qwen3-VL's report says the original allocation "induces an imbalanced frequency spectrum". It interleaves the axes so that, as the cell shows, each axis owns both fast and slow clocks. An interviewer who asks why Qwen3-VL changed its position scheme wants this argument.

The widget draws every pair as a clock, for both released settings, with one row of clocks per axis from fast to slow. Press Play to move two tokens apart along every axis at once. Watch which clocks turn, and which fill in, the sign that a clock has gone all the way round.

Placing a video between text

In a video language model the video tokens sit in one sequence with text, as in "Here is a clip. ⟨video⟩ What does the man do after opening the door?". Every token needs a triple. Qwen2-VL's rule has three parts.

Text tokens get triples with all three coordinates equal to their running position, $(p, p, p)$ . By the theorem, text alone sees 1D RoPE.
A video that starts after $o$ positions have been used gives token $(i, j, k)$ the triple $(o + i,\ o + j,\ o + k)$ . Here $(i, j, k)$ index the grid of tokens the language model reads, after any merging of neighboring tokens.
The text after the video starts at the largest coordinate used so far plus one, $o + \max(n_t, n_h, n_w)$ , and counts up from there.

The third rule is where implementations go wrong. With 1D positions, text after a video of $T$ tokens would start at $o + T$ . With M-RoPE it starts at $o + \max(n_t, n_h, n_w)$ , which is far smaller. A $16\times16\times16$ grid of 4,096 tokens advances the text position by only 16. Qwen2-VL's paper says this "reduces the value of position IDs for images and videos, enabling the model to extrapolate to longer sequences during inference".

Starting at $o + T$ instead is a silent bug. Every score between text and video still depends only on offsets, so nothing crashes. But the offsets are ones the model never saw in training, and its answers about the video quietly get worse.

Interview

3D RoPE for a video followed by text

Write two functions.

mrope_ids(n_t, n_h, n_w, n_text_before, n_text_after) returns an integer array of shape (3, n_text_before + n_t*n_h*n_w + n_text_after) of position triples, one column per token. Rows 0, 1, and 2 hold the time, height, and width coordinates. Follow the three rules above, with the video tokens in time-row-column order. Every grid size is at least 1.
apply_mrope(x, ids, sections, base=10000.0, layout="sectioned") rotates x of shape (T, d_k) using ids of shape (3, T). Pair $j$ is coordinates (2j, 2j+1), rotated counterclockwise by the angle ids[a, t] * base**(-2j/d_k), where $a$ is the axis that owns pair $j$ under the definition's sectioned or interleaved partition. The tuple sections = (s_t, s_h, s_w) counts pairs per axis, with s_t + s_h + s_w = d_k // 2.

import numpy as np

def mrope_ids(n_t, n_h, n_w, n_text_before, n_text_after):
    total = n_text_before + n_t * n_h * n_w + n_text_after
    return np.tile(np.arange(total), (3, 1))           # placeholder: 1D positions in all three rows

def apply_mrope(x, ids, sections, base=10000.0, layout="sectioned"):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Frame indices, seconds, or timestamps

Rule 2 makes the time coordinate a slot index. But how long a slot lasts depends on how the video was sampled. At a sampling rate of $f_{\text{smp}}$ frames per second (§29.1), a slot covers $P_t/f_{\text{smp}}$ seconds. At 2 frames per second with $P_t = 2$ , one slot is one second. At 0.5 frames per second it is four seconds. A model whose time coordinate counts slots cannot tell a slow video from a fast one. Nor can it answer "at what second does the dog jump?" without being told the rate. The Qwen models have tried three answers, each fixing a problem of the one before.

📌 Time in frontier video language models

Qwen2-VL (Wang et al. 2024) samples videos at 2 frames per second and embeds them with $2\times14\times14$ tubelets. It merges each $2\times2$ group of neighboring tokens into one language-model token, so one token covers $2\times28\times28$ pixels. It treats each image as two identical frames, limits each training video to 16,384 tokens by lowering the resolution, and uses slot indices as time coordinates.

Qwen2.5-VL (Bai et al. 2025a) aligns the time coordinate with absolute time and trains with many frame rates. In its released configuration the time coordinate advances by 2 per second of video, whatever the sampling rate.

Qwen3-VL (Bai et al. 2025b) reports that absolute-time coordinates become "excessively large and sparse" for long videos and need training data spread across many frame rates. It replaces them with a text timestamp such as <3.0 seconds> before each temporal patch, written in both seconds and hours-minutes-seconds formats, together with interleaved M-RoPE.

The trade-off is easy to put in numbers. Under absolute time, the time coordinates of an hour-long video span about $2\cdot3600 = 7{,}200$ positions, while its height and width coordinates span only a few dozen, and the text after it starts more than 7,200 positions later. Text timestamps cost a few tokens per temporal patch. In return they put time where a language model reads best, in words, and they let the model write times back in the same format. That is what temporal grounding, finding when something happens in a video, needs (§29.5).

One numerical warning applies to every scheme with large coordinates. The bfloat16 format holds integers exactly only up to 256 (§5.5), so position 3,001 becomes 3,008. The angles must be computed from the coordinates in float32 or better.

🔬 Time tokens in the vocabulary, an older alternative

An older answer keeps time out of the positions and puts it in the vocabulary. Vid2Seq (Yang et al. 2023) gave a T5 encoder-decoder language model extra tokens that stand for moments of the video. One output sequence can then hold both the captions of a video's events and their start and end times, the task called dense video captioning. LITA (Huang et al. 2024) did the same for a chat model.

Relative time tokens. Fix a number of bins $N$ and add $N$ new tokens $\langle 0\rangle, \dots, \langle N - 1\rangle$ to the vocabulary, each with its own learned embedding row (and its own output row, if the input and output embeddings are not tied). For a video of $\ell$ seconds, a time $t\in[0, \ell)$ is written as the token $\langle n\rangle$ with $n = \lfloor Nt/\ell\rfloor$ , the index of the bin of length $\ell/N$ that contains $t$ . The token $\langle n\rangle$ is read back as the center of that bin, $\hat t = (n + \tfrac12)\,\ell/N$ . Because $t$ lies in $[n\ell/N, (n + 1)\ell/N)$ and $\hat t$ is the middle of that interval, the error is at most $\lvert t - \hat t\rvert\le\ell/(2N)$ .

That bound is the whole trade-off. With $N = 100$ , the value both Vid2Seq and LITA use, a 10-second clip is resolved to 0.05 seconds. An hour-long video is resolved only to 18 seconds, coarser than most events a question asks about. And because the scale is relative, $\langle 37\rangle$ means 3.75 seconds in a 10-second clip and 22.5 minutes in an hour-long video. Turning a token into seconds needs the duration, and the model must learn time as the fraction of the video elapsed. The cell checks the bound on random times and prints what one token means at three durations.

Python · cell 8

Why relative bins rather than one token per second? Vid2Seq tested exactly that (its Table 13). It adds $N = 100$ relative time tokens to a 32,128-token vocabulary and compares them with absolute time tokens, where the $k$ -th token is the $k$ -th second. It scores how well the predicted event segments match the true ones with an F1 score, the harmonic mean of precision and recall. On YouCook2, a dataset of cooking videos with timed step captions, F1 is 27.3 with 100 relative tokens, against 12.0 with 100 absolute tokens and 24.3 with 500. On ActivityNet Captions it is 52.4, against 41.5 and 48.6. A hundred absolute tokens cannot name a moment past 100 seconds, while YouCook2's videos run several minutes, so that variant fails for a reason no training can fix. With 500 absolute tokens the range covers most videos, and relative tokens still win, though the paper does not analyze why.

The two papers describe the bins slightly differently. Vid2Seq quantizes a video into $N$ equally spaced timestamps. LITA's text describes $N$ equal chunks, but its conversion formula, $\langle\operatorname{round}(t(N - 1)/\ell) + 1\rangle$ with tokens numbered from 1, rounds a time to the nearest of $N$ equally spaced points from 0 to $\ell$ . Points spaced $\ell/(N - 1)$ apart have worst-case error $\ell/(2(N - 1))$ , against $\ell/(2N)$ for bins. At $N = 100$ the two differ by about 1 percent.

Next to Qwen3-VL's text timestamps, time tokens spend one token per time instead of several, which matters when an answer lists dozens of events. Text timestamps need no new embedding rows, have no bin size and no ceiling, and use number tokens the language model already knows. The newest Qwen models write times as text.

🔬 Sound and pictures on one clock

Most video has sound, and §29.1 counted it in tokens per second. A model that reads both, called an omni model, must also tell the language model which sound goes with which picture. Qwen2.5-Omni (Xu et al. 2025) does it with time-aligned multimodal RoPE, TMRoPE. It is M-RoPE with every time coordinate counted in one unit of 40 milliseconds.

The audio side sets that unit. The audio encoder turns 16 kHz sound into mel-spectrogram frames every 10 milliseconds, short-time spectra on a perceptual frequency scale (§35.3). It downsamples them by 4, so it emits one token per 40 milliseconds, 25 tokens per second of sound. That is close to the 25 to 32 audio tokens per second the Gemini API counts (§29.1).

Here are the rules. Let a video with its sound start after $o$ positions have been used.

Audio. Audio token $r = 0, 1, \dots$ gets the triple $(o + r,\ o + r,\ o + r)$ , three equal coordinates, as for text.
Video. Visual token $(i, j, k)$ of an $n_t\times n_h\times n_w$ grid sampled at $f_{\text{smp}}$ frames per second gets $\big(o + \lfloor 25\,t_i\rfloor,\ o + j,\ o + k\big)$ , where $t_i = iP_t/f_{\text{smp}}$ is the start time of slot $i$ in seconds. One time step is therefore 40 milliseconds at any frame rate.
Interleaving. The tokens are grouped into 2-second chunks by time coordinate, chunk $c$ holding the coordinates from $o + 50c$ to $o + 50c + 49$ . Within each chunk the visual tokens come first and the audio tokens after them.
After. The next text starts at the largest coordinate used plus one, as in rule 3.

In plain words, sound and pictures share one time axis. A visual token and an audio token from the same instant have equal time coordinates, so by the 3D relative property the time term of their score is the zero-offset term, as for two tokens of one frame. Rows and columns are not aligned (an audio token's row and column coordinates equal its time coordinate), just as for text next to an image.

The chunking matters because a language model attends causally, each token seeing only the tokens before it. If all the audio came after all the video, no visual token could attend to any sound. With 2-second chunks, the pictures of each chunk see the sound of every earlier chunk, and a streaming model has heard and seen the first 2 seconds before the next arrive. The price is the size of the coordinates. They grow by 25 per second, so an hour reaches 90,000, 12.5 times Qwen2.5-VL's 2 per second, which makes the float32 warning above a necessity.

Four seconds by hand. Take 4 seconds sampled at 2 frames per second with $P_t = 2$ , so each time slot spans 1 second and $n_t = 4$ . To keep the numbers small, let each slot have a $2\times2$ grid of merged tokens, 16 visual tokens in all. The sound gives $4\cdot25 = 100$ audio tokens. With $o = 0$ , the four slots have time coordinates 0, 25, 50, and 75, and the audio tokens have 0 to 99. Chunk 0 holds coordinates 0 to 49, so it is the 8 visual tokens of slots 0 and 1, then audio tokens 0 to 49. Chunk 1 is the 8 visual tokens of slots 2 and 3, then audio tokens 50 to 99. The visual token at row 0, column 0 of slot 1 is $(25, 0, 0)$ , and the audio token for the same instant is $(25, 25, 25)$ . The largest coordinate used is 99, so the text after starts at 100. The cell builds this sequence and confirms the order, the shared time coordinates, and where the text resumes.

Python · cell 9

Fewer tokens where nothing moves

Everything so far makes the same number of tokens for every clip of a given size, whether it shows a lecture slide that never changes or a football match. §29.1 measured how much consecutive frames repeat, and video codecs exploit exactly that. The tokenizer can do the same. Run-length tokenization (RLT) borrows the idea of run-length encoding, which stores a repeated value once along with how many times it repeats.

Here it is at one location of the frame, over 5 time slots, with a threshold of 0.1. Each tubelet is compared with the one just before it at the same location. Say the differences for slots 1 to 4 are 0.8, 0.3, 0.02, and 0.01. Slots 1 and 2 changed, so they are kept, along with slot 0, which is always kept. Slots 3 and 4 barely changed, so they are dropped, and slot 2 stands for a run of 3 slots. The location costs 3 tokens instead of 5, with run lengths 1, 1, and 3.

📐 Run-length tokenization (RLT)

Let $\mathbf{x}_{i,s}$ be the flattened tubelet at time slot $i$ and spatial slot $s$ , and let $\tau_{\text{rlt}}$ be a threshold. Keep every tubelet of slot 0. For $i\ge1$ , drop $\mathbf{x}_{i,s}$ if it differs from the tubelet at the same spatial slot in the previous time slot by less than the threshold, and keep it otherwise. Each kept tubelet stands for a run, itself plus the dropped tubelets after it at the same spatial slot. Its run length $n_{\text{run}}$ is the distance to the next kept tubelet at that slot, or to the end of the clip. The kept tokens receive their usual position embedding plus a learned embedding of $n_{\text{run}}$ . The variable-length sequences of a batch are packed together rather than padded.

Choudhury et al. (2024) measure the difference as an L1 distance between the first frame of the earlier tubelet and the last frame of the later one, after ImageNet normalization, and typically use $\tau_{\text{rlt}} = 0.1$ . Packing places several sequences in one row with a block-diagonal attention mask, so tokens attend only within their own sequence. It is the masking trick of §4.4.

The cell and the exercise below use a simpler measure, the mean absolute difference over the whole tubelet, with a threshold suited to raw pixels in $[0, 1]$ . The cell runs the rule on two synthetic clips. A static camera watching one moving square keeps a fifth of its tokens. The same scene with a camera panning one pixel per frame keeps all of them, because a pan changes every patch.

Python · cell 10

The pan is the weakness. RLT compares tubelets at a fixed location, so it sees any camera motion as change everywhere. A codec's motion compensation removes exactly this kind of redundancy, and RLT cannot.

A second, subtler weakness is drift. Each tubelet is compared with the one just before it. A slow fade whose every step is below the threshold is dropped entirely, even after the picture has changed completely. Comparing with the last kept tubelet bounds that error, at the cost of a sequential scan along time. Both weaknesses are reasons other ways of cutting tokens stay in use. Masking drops a fixed random fraction regardless of content (§29.4), and token merging averages similar tokens together inside the network (Bolya et al. 2023).

Core

Run-length tokenization

Write run_length_tokenize(patches, thresh). The input patches has shape (n_t, n_s, D), the flattened tubelets by time slot and spatial slot.

Keep every tubelet of slot 0.
For $i\ge1$ , drop tubelet (i, s) if the mean absolute difference between patches[i, s] and patches[i - 1, s] is strictly less than thresh, and keep it otherwise. Compare with the previous time slot, as RLT does, not with the last kept tubelet.

Return (kept, lengths). Here kept is an integer array of shape (K, 2) of (i, s) pairs in time-major order (sorted by i, then by s). The entry lengths[r] is the run length of kept token r, the number of time slots from it to the next kept tubelet at the same s, or to n_t if there is none.

import numpy as np

def run_length_tokenize(patches, thresh):
    n_t, n_s, _ = patches.shape
    kept = np.argwhere(np.ones((n_t, n_s), bool))         # placeholder: keeps everything
    return kept, np.ones(len(kept), dtype=int)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Tubelets at real scale

Two published numbers let us check everything at once. VideoMAE lists the cost of its ViT-B, ViT-L, and ViT-H encoders on 16 frames at 224 pixels with $2\times16\times16$ tubelets as 180, 597, and 1,192 GFLOPs per view. Our count is the tubelet embedding plus, per layer, $24Td^2$ for the projections and the MLP and $4T^2d$ for the attention core. One caution about units. Vision papers report multiply-adds and call them FLOPs (§27.2), so we divide our count by two before comparing.

Python · cell 11

The language-model side runs the same arithmetic with merging. Qwen2-VL's encoder cuts a $448\times448$ frame pair into $32\times32 = 1{,}024$ tubelets, and the $2\times2$ merge passes $16\times16 = 256$ tokens to the language model. At 2 frames per second, one frame pair per second, that is 256 tokens per second of video. The 16,384-token training cap then holds 64 seconds at that resolution, which is why the resolution is lowered for longer videos.

The same merge explains a number in Qwen2-VL's paper, that a $224\times224$ image becomes 66 tokens. That is $(224/28)^2 = 64$ merged tokens plus a start and an end marker. Qwen3-VL's encoder, SigLIP 2 (Tschannen et al. 2025), uses 16-pixel patches with the same depth of 2 and the same merge, so one of its language-model tokens covers $2\times32\times32$ pixels.

The table collects the tokenization choices for a 16-frame clip at 224 pixels and a ViT-B encoder. TimeSformer is a video Transformer that tokenizes each frame on its own (Bertasius et al. 2021).

tokenization	tokens	motion inside a token	ViT-B encoder GFLOPs	used by
per-frame patches, $1\times16\times16$	3,136	none	899	TimeSformer, CLIP-style frame encoders
tubelets, $2\times16\times16$	1,568	2 frames	361	ViViT, VideoMAE, MAE-ST, Qwen2-VL ( $2\times14\times14$ )
tubelets, $4\times16\times16$	784	4 frames	160	ViViT's Table 1 ablation (with 32-frame inputs)
tubelets plus RLT	at most 1,568, content dependent	2 frames	less, by the tokens dropped	Choudhury et al. 2024
tubelets, then a $2\times2$ merge before the language model	392 to the language model	2 frames	361 in the encoder	Qwen2-VL, Qwen2.5-VL, Qwen3-VL (Qwen3-VL's 16-pixel patches give exactly this count)

The second table compares the position schemes, with the parameters they add for that clip at $d = 768$ .

positions	parameters	longer clips	relative in attention	used by
joint learned table	1,204,224	interpolate a 3D table	no	ViViT (initialized by repeating the image table)
separable learned tables	156,672	interpolate the 8 temporal vectors	no	MAE-ST
fixed 1D sines over the raster index	0	extend the table	no	VideoMAE
fixed 3D sines and cosines	0	extend each axis	no, but the table's dot products are	a fixed alternative to learned tables (MAE-ST found fixed sines work about as well as its learned ones)
3D RoPE, sectioned or interleaved	0	positions extrapolate, behavior may not	yes, by the theorem	Qwen2-VL, Qwen3-VL language models
time as absolute seconds or text timestamps	0	any duration and frame rate	yes	Qwen2.5-VL, Qwen3-VL
absolute seconds shared with audio tokens (TMRoPE)	0	any duration, coordinates grow by 25 per second	yes, and sound and picture align in time	Qwen2.5-Omni
relative time tokens, $N$ bins over the duration	$Nd$ new embedding rows, 76,800 at $N = 100$	any duration, but a bin is $\ell/N$ seconds	not a position, time is carried by tokens	Vid2Seq, LITA

The same space-time patching reappears in two later places. Video generators built on diffusion Transformers cut a compressed latent video, rather than pixels, into space-time patches (§34.5). And vision language models merge and pool these tokens further before the language model reads them (§31.5).

In PyTorch

In PyTorch the tubelet embedding is nn.Conv3d with kernel equal to stride, on the layout $(B, C, T_f, H, W)$ . Its weight has shape $(d, C, P_t, P, P)$ , with the channels before time. So its flattening order differs from our channels-last math, and the reshape-plus-matmul version must permute the brick axes to $(C, P_t, P, P)$ to match. The block checks the convolution against that version. Then it inflates a 2D patch embedding both ways and checks the boring-video fixed point on every time slice.

🔥 In PyTorchfor a machine with PyTorch

The Hugging Face implementation of VideoMAE shows the same choices in a released model. Its configuration names the temporal size tubelet_size, and its patch embedding is the Conv3d above. The snippet downloads weights, so it is shown, not run.

⚠️ Where spatiotemporal tokenizers go wrong

Reshaping a clip into rows without the transpose produces rows of the right shape that are not bricks, and the model trains on scrambled inputs without any error. Flattening the brick in one order (channels last) and the weight in another (PyTorch's channels first) does the same, which bites when converting weights between frameworks. Inflating without dividing by $P_t$ makes every first-layer token $P_t$ times too large. A 17-frame clip with $P_t = 2$ silently drops a frame, so check that $T_f$ is a multiple of $P_t$ .

A learned temporal table has no vector for slots it never saw, so longer clips need interpolation and some fine-tuning. Frame-index positions mean different durations at different sampling rates, and a relative time token means different seconds in videos of different lengths. In M-RoPE, two bugs give the model positions it never trained on. One starts the text after a video at the number of video tokens instead of the largest position plus one. The other swaps height and width on a non-square grid. Only a test with a non-square grid and mixed text and video catches them. Large time coordinates must be turned into angles in float32. And RLT treats a camera pan as change everywhere, while silently dropping slow fades.

🧭 What to remember

A tubelet embedding cuts a clip into $P_t\times P\times P$ bricks and multiplies each flattened brick by one shared matrix. It gives $T = \lfloor T_f/P_t\rfloor\lfloor H/P\rfloor\lfloor W/P\rfloor$ tokens, 1,568 for 16 frames at 224 with $2\times16\times16$ .
It equals Conv3d with kernel equal to stride, and it is one matrix multiply after a reshape and a transpose. Its cost, $2\cdot(\text{pixels})\cdot C\cdot d$ , does not depend on the brick shape, while $P_t = 2$ cuts the encoder after it by about 2.5 times.
Inflation ( $\mathbf{W}_{2D}/P_t$ at every tap) and central-frame initialization both reproduce the image model on a boring video, and with joint attention, time-independent positions, and mean pooling so does the whole ViT. Central frame trained best in ViViT (79.2 against 77.6).
Positions come as joint or separable learned tables (separable costs $(n_t + n_s)d$ parameters and resizes along time alone), 3D sinusoids whose dot products split into time, row, and column terms, or 3D RoPE, whose scores depend only on the 3D offset.
M-RoPE deals one ladder of frequencies among three axes, so the partition matters. Sectioning starves width and gives time only fast clocks, and interleaving balances them. Text after a video starts at the largest position plus one. Time has moved from frame indices to absolute seconds to text timestamps, with relative time tokens ( $N$ bins, error up to $\ell/(2N)$ ) as the vocabulary alternative and one 40-millisecond clock for sound and pictures in omni models.
Run-length tokenization drops tubelets that repeat the previous slot, up to 80 percent of the tokens on static video, at the cost of variable lengths and blindness to camera motion.
Next, §29.3 asks how a network should mix these tokens across space and time, from 3D CNNs to divided and factorized attention, with the cost of each derived.

Arnab et al. (2021). ViViT: A Video Vision Transformer. Section 3.2 defines tubelet embedding, Section 3.4 the inflation and central-frame initializations and the repeated position table, and Table 1 compares them.
Carreira and Zisserman (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Section 2.4 introduces inflating 2D filters and the boring-video fixed point.
Tong et al. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. The $2\times16\times16$ cube embedding, the compute of ViT-B to ViT-H on 1,568 tokens, and the from-scratch versus ImageNet baselines. With it, Feichtenhofer et al. (2022), Masked Autoencoders As Spatiotemporal Learners, whose Section 3 gives the patch size and the separable space and time position embeddings.
Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. Section 2.1 introduces M-RoPE, depth-two 3D convolutions, two identical frames per image, and the 16,384-token cap.
Bai et al. (2025b). Qwen3-VL Technical Report. Sections 2.1 and 2.3 explain interleaved MRoPE and text timestamps, and why absolute-time positions were dropped. The time-aligned M-RoPE they replace is in Bai et al. (2025a), Qwen2.5-VL Technical Report, and its audio-aligned form, TMRoPE with 2-second interleaving, in Section 2.2 of Xu et al. (2025), Qwen2.5-Omni Technical Report.
Yang et al. (2023). Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning. Section 3.1 introduces relative time tokens, and Table 13 compares them with absolute ones. Huang et al. (2024), LITA: Language Instructed Temporal-Localization Assistant, brings the same tokens to a chat model.
Wei et al. (2025). VideoRoPE: What Makes for Good Video Rotary Position Embedding?. An analysis of how frequency allocation and layout affect long-video retrieval, with the distractor needle test.
Choudhury et al. (2024). Don't Look Twice: Faster Video Transformers with Run-Length Tokenization. Content-aware token dropping with run-length embeddings and packing.

Made for larger screens

Spatiotemporal tokens

Cutting a clip into bricks

What a token costs, and what it sees

Count the tokens

A tubelet embedding is a 3D convolution

Tubelet embedding without loops

Starting from an image model

Inflate an image model

Telling each token where and when it is

Learned tables, joint and separable

Three-dimensional sines and cosines

Three-dimensional RoPE

Which frequencies each axis gets

Placing a video between text

3D RoPE for a video followed by text

Frame indices, seconds, or timestamps

Fewer tokens where nothing moves

Run-length tokenization

Tubelets at real scale

In PyTorch

Read more

Watch next