Chapter 31 · Section 31.5 · 85 min

High resolution, documents, and video

A vision-language model reads an image by cutting it into small squares, turning each square into a vector, and handing those vectors to a language model as if they were words. This section is about images whose small details carry the answer, such as a scanned invoice, a phone screenshot, a chart, or a video. These are the inputs people send to deployed assistants, and they are where the simplest designs fail. Two numbers decide both the accuracy on such inputs and most of the serving bill. One is how many pixels the image encoder sees. The other is how many vectors the language model reads.

Start from the simplest design, the one §31.1 built. An image encoder is a network that turns an image into vectors. Most open models use a Vision Transformer, a Transformer that cuts the image into squares called patches and treats each patch as a token (§27.1). The usual choice is the one trained by CLIP to match photos with their captions (§30.1). It takes a square of 336 by 336 pixels, with patches of 14 by 14 pixels, so it sees 24 rows of 24 patches and returns 576 feature vectors. A small network called the projector maps each vector into the language model's input space. The 576 results are the visual tokens, and they go into the prompt where words would go. That is the design of LLaVA-1.5 (Liu et al. 2023b).

No later stage can repair that. §31.1 showed that a connector cannot add information that the encoder's features do not carry. The obvious fix is to give the encoder more pixels, and the obvious fix is expensive. At its native resolution the page is $88\times125 = 11{,}000$ patches. The encoder's attention compares every patch with every other, so its cost grows with the square of that number. And every patch that survives becomes a token that the language model carries through every layer and keeps in memory.

So the question of this section is how to get the pixels without paying full price. §31.1 asks three questions of every design in this chapter. Where does the image enter the model? How many visual vectors does it cost? What is trained? Here the image still enters as tokens in the language model's input, and the second question is the subject. Its answer changes the third, because some designs keep the pretrained encoder and others retrain it to read any shape.

We first measure how much resolution matters and what it costs. Then come the two routes the field has taken. One cuts the image into tiles that a fixed encoder can read, and the other teaches the encoder to read any size directly. Along the way we shrink the token count three ways, by merging neighbors, by merging look-alikes, and by dropping tokens inside the language model. We then give positions to a sequence that mixes text, images, and video, turn a video into a budget of frames and tokens, and end with images used as compressed text.

Resolution decides what the model can read

Before spending engineering effort on resolution, it helps to know how much it matters compared with everything else one could change. Three controlled studies agree. For reading fine detail, resolution matters more than almost anything else.

📌 Resolution beats almost everything else

Apple's MM1 study (McKinzie et al. 2024) varied the image encoder. Raising its input from 224 to 336 pixels gave about 3 percent on all its metrics. Doubling the encoder's parameters (from ViT-L to ViT-H) usually gave less than 1 percent. After fine-tuning with sub-images, going from 336 to $1{,}344\times1{,}344$ pixels raised the average score by 15 percent relative. At $1{,}792\times1{,}792$ the average fell slightly.
Gemma 3 (Gemma Team 2025, Table 7) gave the language model 256 image tokens in every run and changed only the encoder's input resolution, average-pooling the encoder's output down to 256. It was a short training run of a 2B model. DocVQA, a benchmark of questions about scanned documents, went from 31.9 at 256 pixels to 45.4 at 448 and 59.8 at 896.
Qwen2-VL-7B (Wang et al. 2024a, Table 7) varied the number of image tokens directly. InfoVQA, a benchmark of questions about infographics, scored 28.85 with 64 tokens per image, 65.72 with 576, 74.99 with 1,600, and 77.27 with 3,136. MMMU, college exam questions with images, stayed between 52.8 and 53.4 throughout.

Read together, the three studies show one pattern. Tasks that need fine detail, such as documents, charts, infographics, and small text, improve enormously with pixels. Tasks that need reasoning about a clear picture barely move. The Gemma result adds a subtler point. The language model received 256 tokens in every run, yet an encoder that saw more pixels before pooling wrote better tokens. So it matters where the resolution enters, not only how many tokens leave.

In practice, when a document model misreads text, the first thing to check is the resolution its encoder saw. An interviewer who asks why a VLM cannot read a receipt wants that answer before any talk of connectors.

What more pixels cost

Here is the bill in plain words. The encoder does work for every patch, and its attention does work for every pair of patches. The language model then does work for every visual token in every layer, and it stores a key and a value for every token in every layer.

Now in symbols. Let an image be resized to $H\times W$ pixels and read by a ViT with patch side $P$ , width $d_{\text{vis}}$ , $L_{\text{vis}}$ layers, and MLP width $d_{\text{ff}}$ . It has $N_p = \lfloor H/P\rfloor\lfloor W/P\rfloor$ patches. §27.2 counted the encoder's forward cost in multiply-adds. One multiply-add is two FLOPs (floating-point operations). Leaving out the small cost of the patch embedding, the cost in FLOPs is

F_{\text{enc}} = L_{\text{vis}}\Big(2N_p\,(4d_{\text{vis}}^2 + 2d_{\text{vis}}d_{\text{ff}}) + 4N_p^2d_{\text{vis}}\Big).

In plain words, the first term is the linear layers, whose cost grows with the pixel area, and the second is attention, whose cost grows with the square of the area.

Many models then merge each $\kappa\times\kappa$ block of neighboring patches into one token, as we derive below. The language model then receives $n_v = N_p/\kappa^2$ visual tokens. §31.1 priced them for a language model with $N$ weights in its $L$ blocks, width $d$ , and $n_{kv}$ key-value heads of width $d_k$ . Prefill, the forward pass over the prompt before the first answer token, costs $2N$ FLOPs per token for the matrix products, plus about $2Ld\,n^2$ for causal attention over $n$ tokens. The KV cache, the keys and values kept for every past token so that later tokens can attend to it, holds $2Ln_{kv}d_k$ numbers per token.

The cell prices the invoice at three resolutions. The encoder is CLIP ViT-L/14 ( $d_{\text{vis}} = 1{,}024$ , 24 layers, $d_{\text{ff}} = 4{,}096$ ). The language model is LLaVA-1.5's Vicuna-7B ( $N = 6.48$ billion, $L = 32$ , $d = 4{,}096$ , $n_{kv} = 32$ , $d_k = 128$ ), with 2 bytes per cached number. The cell also prints the height of a lowercase letter after resizing.

Python · cell 1

Two things stand out. At native resolution the letters are legible, about 10 pixels tall, but the page costs 11,000 tokens. That is 18.5 TFLOPs in the encoder, 174 TFLOPs of prefill in the language model, and 5.4 GiB of cache. And the language model, not the encoder, is the bigger bill, about nine times larger here. So every design below does two things. It gets more pixels into the encoder, and then it reduces the number of tokens that leave it.

There are two routes to the first. The first keeps a pretrained encoder that reads one fixed size and cuts the image into tiles of that size, plus one low-resolution view of the whole image for context. It is cheap to adopt, since the encoder is unchanged. The second retrains the encoder to take any size and aspect ratio directly. That needs positions that work for any grid and batches that mix images of different lengths. Either way, neighboring tokens are usually merged before the language model.

Tiling an image for a fixed-size encoder

Here is tiling at its smallest. A $672\times336$ image, twice as wide as tall, is cut into two $336\times336$ tiles. Each goes through the unchanged CLIP encoder and gives 576 tokens. A third pass encodes the whole image squashed to $336\times336$ . This thumbnail, also called the base image, carries the global layout that neither tile sees alone. That is $3\times576 = 1{,}728$ tokens, three times LLaVA-1.5's budget, and every letter keeps its full size.

📜 From split, encode, and merge to AnyRes

LLaVA-1.5-HD (Liu et al. 2023b, Appendix A.1) did this first in the LLaVA line. It chose one of a fixed set of grids of $224\times224$ tiles with at most six tiles ( $1\times1$ to $1\times6$ , $2\times2$ , $2\times3$ , and their transposes). It encoded each tile with CLIP at 224 pixels, stitched the features back into one map, removed features that covered only padding, appended a row-end token after each row, and added the features of the whole image at 224 pixels. The paper's main text measures that global view on the 7B model. It raised three general benchmarks, GQA from 62.9 to 63.8, MME from 1,425.8 to 1,497.5, and MM-Vet from 31.9 to 35.1. LLaVA-NeXT (Liu et al. 2024, the "LLaVA-1.6" blog) carried the same scheme to 336-pixel tiles under the name AnyRes. Monkey (Li et al. 2023), SPHINX (Lin et al. 2023), and UReader (Ye et al. 2023) used related crops.

A tiling scheme makes three decisions. It picks a grid of tiles, fits the image into that grid, and lays the features out for the language model. The first rule we study picks the grid that keeps the most of the image's own pixels and pads the rest. LLaVA-NeXT, which calls it AnyRes, fixes each decision precisely in its released code.

📐 AnyRes tiling with a base image (LLaVA-NeXT)

Let the image be $W$ pixels wide and $H$ tall. Let the encoder's input side be $S$ (336 for CLIP ViT-L/14 at 336), its patch side $P$ , and $g = S/P$ the patches per tile side (24). A list of candidate resolutions $(w, h)$ , each a multiple of $S$ on both sides, is fixed in advance.

Select. For each candidate, let $s = \min(w/W,\ h/H)$ be the scale at which the image fits inside it. Its effective resolution is $e = \min(\lfloor sW\rfloor\lfloor sH\rfloor,\ WH)$ and its waste is $wh - e$ . Choose the candidate with the largest $e$ , break ties by the least waste, and break any remaining tie by the earlier candidate in the list.
Tile. Resize the image by $s$ , keeping its aspect ratio, pad it centered to $w\times h$ , cut it into $(w/S)(h/S)$ tiles in reading order, and encode each. Separately, resize the whole image (stretched) to $S\times S$ and encode it as the base image.
Lay out. Arrange the tiles' patch features into one map of $R = (h/S)\,g$ rows and $C = (w/S)\,g$ columns. Remove the rows or columns that hold only padding. If $W/H > C/R$ the padding is above and below, and $R - 2\lfloor(R - \lfloor HC/W\rfloor)/2\rfloor$ rows are kept. Otherwise it is left and right, and $C - 2\lfloor(C - \lfloor WR/H\rfloor)/2\rfloor$ columns are kept. Append one learned newline vector after each kept row.

The language model reads the $g^2$ base tokens followed by the rows. With $R'$ rows and $C'$ columns kept,

n_v = g^2 + R'(C' + 1).

In plain words, pick the grid that keeps the most of the image's own pixels, with no credit for blowing the image up. Prefer the smaller grid when two keep the same amount. Give the language model the global view first and then the detail, row by row, with an explicit line break after each row.

Each rule is there for a reason. The cap $e \le WH$ means a small image earns nothing by being upscaled into a large grid, which would cost tokens and show no new detail. Removing the padding rows keeps the padding from costing tokens. The newline vector matters because, once the grid varies, the language model can no longer infer where a row ends from the token count. The projector of §31.1 treats each patch on its own, so nothing else in the sequence marks the layout.

The candidate list is a design choice. The configuration files of the released LLaVA-1.6 checkpoints list five resolutions (width by height), $336\times672$ , $672\times336$ , $672\times672$ , $1{,}008\times336$ , and $336\times1{,}008$ , and the numbers below use those. Notice that $336\times336$ is not on the list, so every image gets at least two tiles. The LLaVA-NeXT blog describes a slightly different list ( $2\times2$ , $1\times\{2, 3, 4\}$ , and $\{2, 3, 4\}\times1$ tiles, up to $1{,}344\times336$ ), so check the configuration of the checkpoint you actually serve.

✏️ Three images through AnyRes, by hand

A $672\times336$ image. The candidate $672\times336$ fits it at $s = 1$ , so $e = 225{,}792 = WH$ , the most any candidate can reach, with no waste. The map is $R = 24$ rows by $C = 48$ columns. The image's aspect ratio $W/H = 2$ equals $C/R$ , so no padding is removed, and $n_v = 576 + 24\cdot49 = 1{,}752$ tokens.

A $300\times900$ receipt. The candidate $336\times1{,}008$ fits it at $s = \min(1.12, 1.12) = 1.12$ and keeps all $270{,}000$ pixels. The candidates $336\times672$ and $672\times672$ both fit it at $s = 0.747$ , as $224\times672$ , which keeps $150{,}528$ . So the receipt gets three tiles stacked vertically, a map of $72\times24$ with nothing to remove, and $n_v = 576 + 72\cdot25 = 2{,}376$ .

A $1{,}920\times1{,}080$ screenshot. The candidate $672\times672$ fits it at $s = 0.35$ , as $672\times378$ , so $e = 254{,}016$ . That beats every other candidate ( $672\times336$ fits it as $597\times336$ , $e = 200{,}592$ ). The map is $48\times48$ , and the screenshot is wider than the map, so $\lfloor 1{,}080\cdot48/1{,}920\rfloor = 27$ rows hold image content. The code removes $\lfloor(48 - 27)/2\rfloor = 10$ padding rows from the top and 10 from the bottom, and $R' = 28$ rows remain. So $n_v = 576 + 28\cdot49 = 1{,}948$ , not the $576 + 48\cdot49 = 2{,}928$ of a full square.

The screenshot shows the trade-off in one line. The grid that keeps the most pixels is a square, and a square grid for a wide image means padding, which the unpadding step then removes. An interviewer who asks for the token count of a given image under a given scheme expects exactly this arithmetic.

Interview

AnyRes grid selection and token count

Write the two halves of the rule in the definition. First, select_best_resolution(size, candidates) takes size = (W, H), width first, and a list of (w, h) candidates, and returns the chosen (w, h). That is the candidate with the largest effective resolution, capped at $WH$ . Ties go to the least waste, and remaining ties to the earliest candidate. Compute $\lfloor sW\rfloor$ and $\lfloor sH\rfloor$ as int(W * s) and int(H * s), as the released code does.

Second, anyres_tokens(W, H, candidates, S=336, P=14, newline=True) returns the number of visual tokens. Count the $g^2$ tokens of the base image, then the tile feature map after its padding rows or columns are removed, then one newline token per kept row when newline is true. Use integer arithmetic for the unpadding.

def select_best_resolution(size, candidates):
    W, H = size
    return min(candidates, key=lambda c: abs(c[0] / c[1] - W / H))   # closest aspect ratio

def anyres_tokens(W, H, candidates, S=336, P=14, newline=True):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Choosing tiles by aspect ratio

A second rule differs on two points. It picks the grid whose shape is closest to the image's shape, rather than the grid that keeps the most pixels. And it stretches the image to fill the grid instead of padding it. InternVL 1.5 (Chen et al. 2024a) introduced it. Its tiles are 448 pixels on a side, and each tile's 1,024 patches are merged into 256 tokens by an operation we derive in the next section.

📐 Closest-aspect-ratio tiling with a thumbnail (InternVL 1.5)

Let the tile side be $S = 448$ and the allowed tile counts $n_{\min} \le ab \le n_{\max}$ (1 to 12 in training). The candidate grids $(a, b)$ , with $a$ tiles across and $b$ down, are all pairs with $n_{\min}\le ab\le n_{\max}$ , sorted by $ab$ . Visit them in that order and keep the one whose aspect ratio $a/b$ is closest to $W/H$ . When a later grid is exactly as close as the current choice (as $2\times2$ is after $1\times1$ , since both have ratio 1), it replaces the current choice if the image area exceeds half the grid's area, $WH > \tfrac12 S^2ab$ . Resize the image to $aS\times bS$ without preserving its aspect ratio, cut it into $ab$ tiles, and if $ab > 1$ append a thumbnail of the whole image at $S\times S$ . Each 448-pixel tile is $32\times32 = 1{,}024$ patches of 14 pixels, merged $2\times2$ into 256 tokens, so

n_v = 256\,\big(ab + \mathbb{1}[ab > 1]\big).

In plain words, choose the grid shaped most like the image, prefer more tiles only when the image is big enough to fill them, and stretch the image to fit.

✏️ The screenshot under the aspect-ratio rule

With $n_{\max} = 12$ there are 35 grids. A $1{,}920\times1{,}080$ screenshot has $W/H = 1.78$ . The closest ratios available are 2 (from $2\times1$ and $4\times2$ ) and 1.5 (from $3\times2$ ), and $|2 - 1.78| = 0.22$ beats $|1.5 - 1.78| = 0.28$ . Between $2\times1$ and $4\times2$ , the image area of $2{,}073{,}600$ exceeds half the area of $4\times2$ , which is $802{,}816$ , so the larger grid wins. That is eight tiles plus a thumbnail, $9\times256 = 2{,}304$ tokens, with the screenshot stretched from aspect ratio 1.78 to 2.

The paper's own example is an $800\times1{,}300$ image, resized to $896\times1{,}344$ , which is six tiles in a $2\times3$ grid. At test time InternVL 1.5 allows up to 40 tiles, which it calls 4K resolution, at a cost of $41\times256 = 10{,}496$ tokens.

The cell compares the two rules on five documents. "Read at" is the linear scale at which the encoder sees the image, so 1.00 means native resolution and the letters keep their size. When the image is stretched, the scale is the geometric mean of the two axes.

Python · cell 2

The two rules disagree in instructive ways. On the panorama, the pixels-kept rule chooses three tiles in a row and reads the image at a quarter of its size. The aspect-ratio rule chooses $4\times1$ , which costs five views, and still shrinks the panorama to less than half. On the A4 scan, the aspect-ratio rule's six tiles read the page at about three quarters of its size, and the pixels-kept rule's square grid at about two fifths. Neither rule reads every document at native size, which motivates the second route.

The widget runs both rules, the plain 336-pixel square, and the native-resolution rule from later in this section, on any image. Pick a document, choose a scheme, and watch the tiles, the padding, and the token count.

The count jumps. The A4 page stays at six tiles and 1,792 tokens up to a limit of 34. At 35 the $5\times7$ grid becomes allowed, its shape is closer to the page's, and the cost leaps to 9,216 tokens, with the page now upscaled to 1.8 times its size. On the panorama, the plain square pads three quarters of its area, the pixels-kept grid pads a quarter, and the aspect-ratio rule pads nothing, because it stretches instead. So raising the tile limit often changes nothing for a given page and then, at one value, multiplies its cost. A serving team should set the limit with the expected page shapes in view.

Thumbnails, tile tags, and seams

Tiles create three problems of their own. No single tile shows the whole scene. The language model does not know how the tiles were arranged. And a word or an object cut by a tile border is split between two encodings that never saw each other. Each problem has a fix whose gain was measured.

📌 What thumbnails, tags, and overlap buy

A global view. In LLaVA-1.5-HD's ablation above, adding the whole image at 224 pixels raised MME from 1,425.8 to 1,497.5 on the 7B model.
Tile tags. NVLM (Dai et al. 2024, Table 1) put a text tag before each tile's 256 tokens, <tile_1>, <tile_2>, and so on, with <tile_global> before the thumbnail. On NVLM-D with Yi-34B, DocVQA was 52.9 with a single $448\times448$ view, 80.2 with dynamic tiles and no tags, and 87.4 with these one-dimensional tags. ChartQA, questions about charts, went from 64.8 to 76.1 to 81.1. The one-dimensional tags matched or beat two-dimensional grid tags and bounding-box tags (86.7 each on DocVQA), even though they do not say whether a grid is $2\times3$ or $3\times2$ . The authors attribute this to better generalization at test time.
Overlapping crops. Molmo (Deitke et al. 2024) moved neighboring crops closer together so they overlap by 4 patches (56 pixels). It passed to the language model only the features outside the overlap, so they still tile the image exactly, but every border patch has seen its neighbors. Its 11-benchmark average was 62.8 with a single crop, 75.7 with crops and no overlap, and 76.9 with overlap.

A favorite interview follow-up is why a thumbnail is needed when the tiles already cover every pixel. The answer is that each tile is encoded alone, so only the thumbnail lets any token see the whole layout.

🔬 Two more tiling variants in wide use

LLaVA-OneVision (Li et al. 2024) runs AnyRes with SigLIP, a CLIP-style encoder, at 384 pixels, 729 tokens per crop, with grids from $1\times1$ up to $6\times6$ crops plus the base image. It caps a single image at a threshold of $729\times10 = 7{,}290$ tokens. When the $ab + 1$ views of an $a\times b$ grid would exceed the threshold, it shrinks every view's feature map by bilinear interpolation to an equal share, the threshold divided by $ab + 1$ .

Gemma 3 (Gemma Team 2025) has a fixed $896\times896$ encoder. Only at inference, and only when needed, it applies a step called Pan and Scan, which cuts a non-square or large image into equal non-overlapping crops, each resized to $896\times896$ . On the 27B model this raised InfoVQA from 59.4 to 76.4 and DocVQA from 85.6 to 90.4.

Merging neighbors with pixel shuffle

A 448-pixel tile is 1,024 patches, so twelve tiles would be more than 12,000 tokens. The common fix is to merge each $2\times2$ block of neighboring feature vectors into one token by stacking their numbers end to end. Nothing is averaged away. Four vectors of width $c$ become one vector of width $4c$ , and the projector that follows decides what to keep. The operation is called pixel shuffle, or space-to-depth, because it moves spatial neighbors into the depth, the channel dimension. InternVL, Qwen2-VL, Kimi-VL, and NVLM all use it.

✏️ A 4 by 4 grid, by hand

Let $\mathbf{X}$ be a $4\times4$ grid of one-number tokens holding $1, 2, \dots, 16$ in reading order, and merge $2\times2$ . The top-left block holds $1, 2$ above $5, 6$ , so the first merged token is $\mathbf{Y}_{0,0} = (1, 2, 5, 6)$ . Likewise $\mathbf{Y}_{0,1} = (3, 4, 7, 8)$ , $\mathbf{Y}_{1,0} = (9, 10, 13, 14)$ , and $\mathbf{Y}_{1,1} = (11, 12, 15, 16)$ . Sixteen tokens of width 1 have become four of width 4.

Now apply a linear layer with weights $\mathbf{W} = (w_1, w_2, w_3, w_4)^\top$ . The output at $(0, 0)$ is $w_1 + 2w_2 + 5w_3 + 6w_4$ . That is the $2\times2$ kernel $\left(\begin{smallmatrix} w_1 & w_2\\ w_3 & w_4\end{smallmatrix}\right)$ placed on the top-left block, a convolution. Average pooling is the special case $w_1 = w_2 = w_3 = w_4 = \tfrac14$ , which gives 3.5.

Here is the general rearrangement, with $\kappa$ for the merge factor.

📐 Pixel shuffle (space-to-depth) with factor κ

Let $\mathbf{X}\in\R^{h\times w\times c}$ be a grid of feature vectors with $h$ and $w$ divisible by $\kappa$ . Space-to-depth gives $\mathbf{Y}\in\R^{(h/\kappa)\times(w/\kappa)\times\kappa^2c}$ with

Y_{a,\,b,\,(i\kappa + j)c + m} = X_{a\kappa + i,\ b\kappa + j,\ m}, \qquad 0\le i, j < \kappa,\ \ 0\le m < c .

So output token $(a, b)$ is the concatenation of the $\kappa^2$ input vectors of block $(a, b)$ in reading order, row $i$ then column $j$ . Depth-to-space is the inverse rearrangement. A pixel-shuffle connector applies space-to-depth and then an MLP from $\R^{\kappa^2c}$ to the language model's width $d$ .

The channel order is a convention. PyTorch's F.pixel_unshuffle uses the channel-major order $m\kappa^2 + i\kappa + j$ on channels-first tensors, and published implementations differ. That is harmless as long as training and inference use the same one. Space-to-depth only permutes the entries, so by itself it loses nothing. What it trades is width for length. The interesting fact is what a linear layer after it computes, which the example already showed for one block.

∴ Pixel shuffle followed by a linear map is a strided convolution

Let $\mathbf{W}\in\R^{\kappa^2c\times d'}$ and define the kernel $\mathbf{K}\in\R^{\kappa\times\kappa\times c\times d'}$ by $\mathbf{K}_{i,j,m,:} = \mathbf{W}_{(i\kappa + j)c + m,\,:}$ . Then for every block $(a, b)$ ,

\big(\mathbf{Y}\mathbf{W}\big)_{a,b,:} = \sum_{i=0}^{\kappa-1}\sum_{j=0}^{\kappa-1}\sum_{m=0}^{c-1} X_{a\kappa+i,\ b\kappa+j,\ m}\ \mathbf{K}_{i,j,m,:},

which is a convolution of $\mathbf{X}$ with a $\kappa\times\kappa$ kernel and stride $\kappa$ . Conversely, every such convolution is space-to-depth followed by a linear map. The proof is in the derivation below.

∂ Proof that shuffle then linear is a strided convolution

By definition of the matrix product, $(\mathbf{Y}\mathbf{W})_{a,b,:} = \sum_{q=0}^{\kappa^2c-1}Y_{a,b,q}\mathbf{W}_{q,:}$ . Every index $q$ in that range is written uniquely as $q = (i\kappa + j)c + m$ with $0\le m<c$ and $0\le i, j<\kappa$ . Indeed $m = q \bmod c$ , and $i\kappa + j = \lfloor q/c\rfloor$ is a number below $\kappa^2$ whose base- $\kappa$ digits are $i$ and $j$ . Re-index the sum by $(i, j, m)$ , and substitute $Y_{a,b,q} = X_{a\kappa+i,\,b\kappa+j,\,m}$ and $\mathbf{W}_{q,:} = \mathbf{K}_{i,j,m,:}$ . That gives the right side. The map from $\mathbf{W}$ to $\mathbf{K}$ is a bijection between the two shapes, so the converse holds too. $\blacksquare$

In plain words, a pixel-shuffle connector is a learned $2\times2$ convolution with stride 2 over the feature map, followed by the rest of the MLP. It is the same identity as the patch embedding of §27.1, which is a convolution whose kernel size equals its stride. Here it is applied one level up, to features instead of pixels.

The example's last remark generalizes. Average pooling over $\kappa\times\kappa$ blocks is space-to-depth followed by the fixed matrix $\tfrac{1}{\kappa^2}[\mathbf{I}_c; \dots; \mathbf{I}_c]$ , a linear map of rank $c$ from $\R^{\kappa^2c}$ . So pooling is the pixel-shuffle connector with its first layer frozen to an average. It keeps the mean of each block and discards everything else. A learned pixel-shuffle MLP can keep whatever the loss needs, at the price of a first layer $\kappa^2$ times wider. That is why most open VLMs that merge tokens stack them rather than pool them, and it is the comparison an interviewer wants when asking how pixel shuffle differs from pooling.

Core

Pixel shuffle and its inverse

Write three functions.

space_to_depth(X, kappa) takes X of shape (h, w, c) and returns shape (h // kappa, w // kappa, kappa * kappa * c), with the definition's channel order $(i\kappa + j)c + m$ . It raises ValueError if h or w is not divisible by kappa.
depth_to_space(Y, kappa) is its exact inverse.
linear_to_kernel(Wm, kappa, c) converts the weight of a linear layer applied after space_to_depth into the kernel of the equivalent stride-kappa convolution. Wm has shape (kappa * kappa * c, d_out), and the kernel K has shape (kappa, kappa, c, d_out). The test is that space_to_depth(X, kappa) @ Wm equals $\sum_{i,j,m}X_{a\kappa+i,\,b\kappa+j,\,m}K_{i,j,m,:}$ at every output position $(a, b)$ .

import numpy as np

def space_to_depth(X, kappa):
    h, w, c = X.shape
    return X.reshape(h // kappa, w // kappa, kappa * kappa * c)

def depth_to_space(Y, kappa):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

The cell checks the theorem and the pooling remark numerically, with the convolution written as a direct loop over output positions.

Python · cell 3

Other compressors fit the same picture. Gemma 3's encoder reads $896\times896$ pixels as $64\times64$ patches and average-pools them $4\times4$ to 256 tokens. LLaVA-OneVision shrinks each video frame's $27\times27$ grid of SigLIP features to $14\times14 = 196$ tokens by bilinear interpolation. DeepSeek-OCR (Wei et al. 2025) puts a 16-times compressor inside the encoder. Two $3\times3$ convolutions with stride 2 take a $1{,}024\times1{,}024$ page from 4,096 patch tokens to 256. A third option compresses the image into a few dozen learned query vectors that attend to the features, as the Q-Former of §31.3 does. The evidence there applies here too. At a fixed token count the connector's design matters little, and the count and the resolution matter a lot.

Merging by content instead of by position

Every compressor above merges by position. It combines the same $\kappa\times\kappa$ blocks whatever the image shows, so a blank sky costs as many tokens as a page of dense text. The patch merging between the stages of a Swin Transformer (§27.5) is the same position-based operation, a $2\times2$ space-to-depth followed by a LayerNorm and a linear map. A third family merges by content instead. It finds tokens that carry nearly the same information and averages them, wherever they sit. Token Merging, or ToMe (Bolya et al. 2023), was designed to speed up a trained ViT without retraining it, and later methods reuse its matching rule.

§27.5 defines ToMe's matching rule, called bipartite soft matching, and measures it. In brief, each block splits its tokens into two alternating halves. Every token of the first half finds its most similar partner in the second half, by the cosine of their attention keys. The $r$ best-matched pairs are merged into their average, so the block ends with $r$ fewer tokens. A merged token remembers how many patches it stands for. That count weights it in later averages and in later attention. Off the shelf this doubles the throughput of ViT-L and ViT-H for a few tenths of a point of ImageNet-1k accuracy. Its strength is that it adapts to the image. A uniform background collapses into a few tokens, and a detailed region is left nearly untouched.

What matters for a vision-language model is where the merged tokens come from. The cell runs one step on a synthetic $24\times24$ grid whose top half is a flat sky and whose bottom half is busy. It uses the features themselves as keys and merges $r = 144$ tokens.

Python · cell 4

All 144 merges fall in the sky, as they should. But 92 percent of them join patches that are not neighbors, some 22 patches apart. That is the weakness for a vision-language model. A merged token can stand for patches from opposite sides of the image, so it has no single row and column. It cannot get the 2D positions that the language model uses later in this section. The output is no longer a grid that pixel shuffle or newline tokens can act on, and an answer that must point at a region has no box to point at. Bolya et al. make the same point for the ViT itself. There the only spatial signal a merged token keeps comes from the position embeddings that were averaged into it. An interviewer comparing merging by position with merging by content wants to hear exactly this trade, adaptivity against a grid.

🔬 LLaVA-PruMerge, content merging inside a VLM

LLaVA-PruMerge (Shang et al. 2024) keeps positions intact by merging into tokens that survive. It looks at the attention from CLIP's class token, the extra token whose output summarizes the image, in the penultimate layer. It keeps the tokens whose attention lies above the upper interquartile fence $Q_3 + 1.5(Q_3 - Q_1)$ , a standard outlier test. It then replaces each kept token by a weighted average of its $k$ most similar tokens, with similarity measured on the last layer's keys and the weights given by class attention. That keeps 5.5 percent of LLaVA-1.5's 576 tokens on average, about 32, and the cost shows. With the 7B model fine-tuned this way, VQAv2 (general visual questions) fell from 78.5 to 72.0. POPE, a yes-or-no test of whether the model invents objects (§31.6), fell from 85.9 to 76.3. The variant PruMerge+ adds a spatially uniform sample of tokens and keeps 25 percent, with VQAv2 at 76.8 and POPE at 84.0. That is the same lesson about coverage that the pruning results below teach.

Reading images at their own size

Tiling is a workaround for an encoder that only knows one size. The cleaner design is an encoder that reads the whole image at its own aspect ratio and close to its own resolution. Two obstacles stand in the way. Positions must work for any grid, and a batch must hold images whose patch counts differ by a factor of forty or more.

The first obstacle was handled in §27.1. A learned table of positions is trained for one grid and must be interpolated for any other. Rotary position embedding, or RoPE, needs no table. It rotates pairs of coordinates of each query and key by angles proportional to the token's position, so an attention score depends only on how far apart two tokens are (§5.5). Axial 2D RoPE rotates half of the pairs by angles set by the patch's column and the other half by angles set by its row. Every score then depends only on the row and column offsets between two patches, whatever the grid. Qwen2-VL replaced its encoder's absolute positions with 2D RoPE. Kimi-VL and Qwen3-VL kept SigLIP's learned positions, interpolated to each grid, and added 2D RoPE on top.

With positions settled, the image only has to be resized to a whole number of tokens. In plain words, the rule rounds each side to a multiple of 28 pixels, the side of one merged token, and scales the image down or up only if its area falls outside fixed limits. Qwen2-VL introduced it, and its successors kept it.

📐 Native-resolution resizing (Qwen2-VL)

Let the patch side be $P = 14$ and the merge factor $\kappa = 2$ , so each language-model token covers $f = P\kappa = 28$ pixels on a side, and fix pixel limits $p_{\min}$ and $p_{\max}$ . Round each side to the nearest multiple of $f$ , $\bar H = \max(f, f\cdot\operatorname{round}(H/f))$ , and likewise $\bar W$ . If $\bar H\bar W > p_{\max}$ , shrink with $\beta = \sqrt{HW/p_{\max}}$ and round down, $\bar H = f\lfloor H/(\beta f)\rfloor$ , $\bar W = f\lfloor W/(\beta f)\rfloor$ . If $\bar H\bar W < p_{\min}$ , grow with $\beta = \sqrt{p_{\min}/(HW)}$ and round up, $\bar H = f\lceil H\beta/f\rceil$ , $\bar W = f\lceil W\beta/f\rceil$ . The encoder reads $(\bar H/P)(\bar W/P)$ patches, and after the $2\times2$ merge the language model reads

n_v = \frac{\bar H}{f}\cdot\frac{\bar W}{f} + 2

tokens, the 2 being the <|vision_start|> and <|vision_end|> markers.

✏️ Three images through the native-resolution rule

The released preprocessor sets $p_{\min} = 3{,}136 = 4\cdot28^2$ and $p_{\max} = 12{,}845{,}056 = 16{,}384\cdot28^2$ , so an image costs between 4 and 16,384 tokens plus the two markers.

A $224\times224$ image is $8\times8 = 64$ tokens, 66 with the markers, the number the Qwen2-VL paper gives.
A $1{,}920\times1{,}080$ screenshot rounds to $1{,}932\times1{,}092$ , a grid of $69\times39 = 2{,}691$ tokens.
The A4 scan rounds to $1{,}232\times1{,}764$ , which is $44\times63 = 2{,}772$ tokens, read at essentially its native size.

The second obstacle is batching. A $224\times224$ thumbnail is 256 patches and a scanned page is 11,000. Padding every image in a batch to the longest would waste most of the compute, and resizing them all to one size is what we are trying to avoid. The answer is the one language models already use for documents of different lengths, packing (§14.3). Put several whole images one after another in each row of the batch, and stop each image's patches from attending to the others. NaViT (Dehghani et al. 2023) made this work for a ViT.

📐 Patch n' Pack (NaViT)

Fix a row length $T_{\text{row}}$ . Each row of a batch holds the patch sequences of several whole images placed one after another, followed by padding. Give every slot $t$ the index $\text{id}_t$ of its image, or $-1$ for padding. Then

Masked self-attention. Slot $t$ may attend to slot $u$ only if $\text{id}_t = \text{id}_u \ge 0$ , a block-diagonal mask (§4.4).
Masked pooling. Any per-image output (a pooled embedding for a contrastive loss, or a classification) averages only that image's slots.
Positions per image. Each patch's position is its (row, column) within its own image. It is encoded with factorized embeddings $\phi_y(r) + \phi_x(c)$ , with fractional coordinates $r/H_p$ and $c/W_p$ for an image of $H_p\times W_p$ patches, or with 2D RoPE.
Optionally, token dropping per image. A different random fraction of each image's patches can be dropped during training.

NaViT packed greedily and found that typically less than 2 percent of tokens were padding. Trained this way on mixed resolutions, NaViT matched the best fixed-resolution ViT of its comparison with four times less pretraining compute. The main reason is that packing and token dropping let it see about five times more images for the same budget. What packing costs in attention is a short calculation.

∴ Attention cost of a packed row

Let a row hold $k$ images with $n_1, \dots, n_k$ patches, and $n = \sum_i n_i$ . With the block-diagonal mask, the useful attention scores number $\sum_in_i^2$ , and

\frac{n^2}{k}\ \le\ \sum_{i=1}^k n_i^2\ \le\ n^2,

with equality on the left exactly when all images have the same size. A dense attention kernel given the mask still computes all $n^2$ scores (plus those of the padding), so up to a fraction $1 - 1/k$ of its attention work is thrown away. A variable-length kernel computes only $\sum_in_i^2$ . It receives the image boundaries as a list of cumulative lengths, called cu_seqlens in FlashAttention (§12.4). The linear layers cost the same per token whether packed or not.

∂ Proof of the bounds

The upper bound holds because $n^2 = \sum_in_i^2 + \sum_{i\ne j}n_in_j$ and the cross terms are nonnegative. For the lower bound, apply Cauchy-Schwarz to $(n_1, \dots, n_k)$ and $(1, \dots, 1)$ . It gives $n = \sum_in_i\cdot1 \le \sqrt{\sum_in_i^2}\sqrt{k}$ , and squaring gives $n^2 \le k\sum_in_i^2$ . Equality holds exactly when the two vectors are parallel, that is, when all $n_i$ are equal. $\blacksquare$

In plain words, packing is free for the MLPs and costs only the attention between images that the mask throws away, and a variable-length kernel does not even compute that. The cell packs a mixed batch of documents and photos, resized by the native-resolution rule, and compares the work.

Python · cell 5

The cell sorts the images by size before first-fit packing, a common improvement that the exercise below leaves out. In the two rows that hold several images, a dense masked kernel would waste 34 and 43 percent of its attention work on pairs of patches from different images. Padding every image to the longest would leave 59 percent of the slots empty. In an interview about batching mixed-resolution images, those two numbers are the case for packing and for variable-length kernels.

Interview

Pack images of any shape

Write pack_images(grids, row_len), where grids is a list of (gh, gw) patch grids, one per image. Place the images in the given order. Put each one into the first existing row with room for all gh * gw of its patches, and open a new row when none has room. That is first fit, with no splitting and no sorting. Within a row, images sit one after another in the order they were placed, followed by padding. Raise ValueError if an image has more than row_len patches. Return three arrays.

ids, shape (R, row_len), the image index of each slot, or -1 for padding.
pos, shape (R, row_len, 2), the (row, column) of each patch within its own image in reading order, or (-1, -1) for padding.
mask, shape (R, row_len, row_len), boolean, True where slot t may attend to slot u, that is, where both belong to the same image. Padding attends to nothing.

Then write masked_mean_pool(Hs, ids, n_images). It takes token states Hs of shape (R, row_len, d) and returns shape (n_images, d), the mean of each image's slots.

import numpy as np

def pack_images(grids, row_len):
    n = [gh * gw for gh, gw in grids]
    ids = np.concatenate([np.full(k, i) for i, k in enumerate(n)])
    ids = np.concatenate([ids, -np.ones(-len(ids) % row_len, int)]).reshape(-1, row_len)   # split images across rows
    mask = ids[:, :, None] >= 0
    return ids, np.zeros(ids.shape + (2,), int), np.broadcast_to(mask, ids.shape + (row_len,))

Tests, verified solutions, and AI help, open once you add your API key.

Native-resolution encoders in practice

A native-resolution encoder pays the quadratic attention of §27.2 on the whole image. At 16,384 merged tokens it sees 65,536 patches. The usual fix is window attention, in which most layers let each patch attend only to patches in a small window around it. Qwen2.5-VL (Bai et al. 2025a) uses windows of at most $112\times112$ pixels, $8\times8$ patches, in all but four of its encoder's 32 layers. For an $896\times896$ image of 4,096 patches, a windowed layer scores $4{,}096\cdot64$ pairs instead of $4{,}096^2$ , which is 64 times fewer. Its cost grows linearly with the image area, and the four full-attention layers still let information cross the whole image.

🔬 Five native-resolution encoders

Qwen2-VL (Wang et al. 2024a) uses a 675M-parameter ViT with 2D RoPE, packs images of different sizes into one sequence at inference, and merges $2\times2$ neighbors with an MLP.
Qwen2.5-VL (Bai et al. 2025a) uses window attention in all layers but 7, 15, 23, and 31 (counting from 0), which keep full attention. It also states bounding boxes in the image's actual pixel coordinates rather than normalized ones, so the model learns scale.
Pixtral 12B (Agrawal et al. 2024) trained a 400M-parameter encoder from scratch with 16-pixel patches, 2D RoPE that alternates rows and columns pair by pair, and a block-diagonal mask for packed images. It puts a break token between image rows and an end token after the image, so two images with the same number of patches but different shapes are told apart.
Kimi-VL (Kimi Team 2025) continued training SigLIP-SO400M into an encoder it calls MoonViT, with NaViT packing and variable-length FlashAttention. It then merges $2\times2$ by pixel shuffle before a two-layer MLP.
Qwen3-VL (Bai et al. 2025b) uses SigLIP-2 SO-400M (SigLIP-2 Large for its 2B and 4B models) with 16-pixel patches, so a merged token covers 32 pixels. It adds a step called DeepStack, which takes features from three intermediate encoder layers (8, 16, and 24 in the released 8B configuration) and adds them, through their own mergers, to the hidden states of the first three language-model layers.

The cell runs one A4 page and one screenshot through all four schemes we have defined. It prints the tokens, the scale at which each is read, and how many pixels of the image, at the scale the encoder saw it, each token carries.

Python · cell 6

Only the native-resolution rule reads both at full size, and it does so at about 2,700 tokens, one token per $28\times28$ square of the source. Tiling with a padded square grid spends 70 to 80 percent as many tokens and reads both at well under half their size. That is the practical case for native resolution when you control the encoder's training.

Positions in the language model

Inside the encoder, 2D positions are now standard. The language model has the opposite problem. It reads text, images, and video in one sequence with one-dimensional RoPE, which gives each token a single position number and makes every attention score depend only on the difference of two such numbers. A flattened image loses its geometry under that scheme. In the $69\times39$ screenshot, token $t$ and token $t + 69$ are vertical neighbors, yet RoPE sees them as 69 positions apart.

Multimodal RoPE, or M-RoPE, from Qwen2-VL fixes this. §29.2 defined and analyzed it, and here is the idea again. Each token gets three coordinates, for time, height, and width, and each frequency pair of a query or key rotates with one of them. §29.2 proved the two facts we need. The score between two tokens depends only on their offsets along the three axes. And a token whose three coordinates are equal is rotated exactly as 1D RoPE would rotate it, so text alone behaves as in any language model.

What remains is the bookkeeping for a whole conversation, with several images and videos between pieces of text. §29.2 placed one video between two pieces of text. Here is the general rule.

📐 M-RoPE position ids for a multimodal sequence (Qwen2-VL and Qwen2.5-VL)

Keep a cursor $p$ , starting at 0, and visit the segments in order. Every token gets a triple $(p^{\text{t}}, p^{\text{h}}, p^{\text{w}})$ .

A text token (including the vision start and end markers) gets $(p, p, p)$ , and $p$ increases by 1.
An image whose merged grid is $g_h\times g_w$ gives its token in row $r$ and column $c$ the triple $(p,\ p + r,\ p + c)$ , in reading order. The time coordinate is constant.
A video of $g_t$ temporal steps, each a $g_h\times g_w$ grid, gives the token at step $i$ , row $r$ , and column $c$ the triple $(p + \lfloor\tau i\rfloor,\ p + r,\ p + c)$ . In Qwen2-VL $\tau = 1$ , so time counts steps. In Qwen2.5-VL $\tau$ is the number of time ids per temporal step, $\tau = 2\times(\text{seconds per step})$ , so time ids follow the clock at two ids per second whatever the sampling rate. (Here $\tau$ is a local symbol, not a temperature.)
After an image or a video, the cursor moves to the largest id used so far plus one.

The last rule is what keeps positions small. An image moves the cursor by $\max(g_h, g_w)$ instead of by its token count, and a video by $\max(\lfloor\tau(g_t - 1)\rfloor + 1,\ g_h,\ g_w)$ . The $69\times39$ screenshot advances the text position by 69, not by 2,691. A conversation with ten screenshots uses about 700 positions for its images instead of 27,000. The Qwen2-VL paper says this reduces the values of the position ids of images and videos and lets the model extrapolate to longer sequences at inference. Its Table 8 compares M-RoPE with 1D RoPE on a small model (Qwen2-1.5B with a ViT-L). The gains are largest on two video question-answering benchmarks, NextQA (46.0 against 43.9) and STAR (57.9 against 55.5).

✏️ Position ids by hand

Three text tokens, an image whose merged grid is $2\times3$ , then two text tokens. The text gets $(0,0,0)$ , $(1,1,1)$ , $(2,2,2)$ , so $p = 3$ . The image's six tokens are $(3, 3, 3)$ , $(3, 3, 4)$ , $(3, 3, 5)$ , $(3, 4, 3)$ , $(3, 4, 4)$ , $(3, 4, 5)$ . The largest id is 5, so the cursor moves to 6, and the last text tokens are $(6,6,6)$ and $(7,7,7)$ . With 1D positions the image would have occupied 3 to 8, and the text would be at 9 and 10.

Now replace the image by a video of three steps of $1\times2$ tokens, sampled at 2 frames per second with two-frame temporal patches. Each step is one second, so $\tau = 2$ in Qwen2.5-VL. Starting at $p = 3$ , its time ids are $3, 5, 7$ (two tokens each), its height ids are all 3, and its width ids alternate 3 and 4. The text after it starts at $7 + 1 = 8$ . Sampled at 1 frame per second instead, each step is two seconds, $\tau = 4$ , and the same three steps get time ids $3, 7, 11$ . The gap between ids now says how much time passed, which is the point of Qwen2.5-VL's change.

Interview

M-RoPE position ids for a conversation

Write mrope_ids(segments), which returns the position triples of the definition for a whole conversation. Each segment is one of three kinds.

("text", n) is n text tokens.
("image", gh, gw) is an image with a merged grid of gh rows and gw columns.
("video", gt, gh, gw, tau) is gt temporal steps of a gh by gw grid, with tau time ids per step.

Return an integer array of shape (3, T), with rows $p^{\text{t}}$ , $p^{\text{h}}$ , $p^{\text{w}}$ and one column per token. Order image and video tokens by step, then row, then column, and give step $i$ the time id $p + \lfloor\tau i\rfloor$ . An empty list returns shape (3, 0).

import numpy as np

def mrope_ids(segments):
    T = 0
    for s in segments:
        T += s[1] if s[0] == "text" else int(np.prod(s[1:4] if s[0] == "video" else s[1:3]))
    return np.tile(np.arange(T), (3, 1))                 # 1D positions in all three rows

Tests, verified solutions, and AI help, open once you add your API key.

The widget lays out the ids of a short conversation. Switch to 1D positions to see how far an image pushes the text that follows it.

One bug here crashes nothing. Suppose the code moves the cursor past an image by its token count, the 1D habit, instead of to the largest id plus one. Every score still depends only on offsets, but the offsets between the text and the image are ones the model never saw in training, and its answers quietly get worse. §29.2 shows the same bug for video. With many screenshots in one conversation, the error grows with every image and shifts every later turn.

Qwen3-VL has since changed the video rule. It replaces absolute-time ids with text timestamps and interleaves the three axes across the frequency pairs, for the reasons §29.2 gives. Its rule for images is the one in the definition.

Video as a frame and token budget

§29.1 priced video in tokens, a count that grows linearly with duration, frame rate, and pixel area. §29.2 showed how a 3D patch spanning two consecutive frames, called a temporal patch, halves that count. A vision-language model has to turn those formulas into one decision per request. How many frames should it take, and at what resolution, under a cap on tokens?

Qwen2-VL reads video at 2 frames per second with two-frame temporal patches and a training cap of 16,384 tokens per video (§29.2 gives the details). One result is new here. In its Figure 5 the 72B model holds its Video-MME accuracy as inference sequences grow past that training length, up to 80K tokens, which the paper credits to M-RoPE (Wang et al. 2024a).

Here is a planner in that style, written as a rule we can test. In plain words, it decides the frames first, divides the token budget equally among the temporal steps, and lets the resolution be whatever that leaves. The released preprocessing utilities follow the same logic with more options.

📐 A frame and token budget

Let a video last $\ell$ seconds with $N_{\text{src}} = \lfloor\ell f_{\text{src}}\rfloor$ frames at its source rate $f_{\text{src}}$ . It is sampled at a target rate $r_{\text{fps}}$ , with temporal patches of 2 frames, a token cap $N_{\max}$ , a frame cap $F_{\max}$ , and a merged-token side of $f = 28$ pixels.

Frames. $F = 2\lfloor\min(\ell r_{\text{fps}},\ F_{\max},\ N_{\text{src}})/2\rfloor$ , and at least 2. The frame indices are $\operatorname{round}(\operatorname{linspace}(0, N_{\text{src}} - 1, F))$ , evenly spaced from the first frame to the last, and the timestamps are the indices divided by $f_{\text{src}}$ .
Budget per step. The video has $F/2$ temporal steps, and each may spend $n_{\text{step}} = \lfloor N_{\max}/(F/2)\rfloor$ tokens. If $n_{\text{step}} < 4$ , keep only $\lfloor N_{\max}/4\rfloor$ steps (so $F$ is twice that), re-space the frames, and recompute $n_{\text{step}}$ .
Resolution. Resize each frame with the native-resolution rule, with $p_{\max} = n_{\text{step}}f^2$ and $p_{\min} = 4f^2$ .
Tokens. The video costs $(F/2)(\bar H/f)(\bar W/f)$ tokens, which is at most $N_{\max}$ (before the two vision markers and any timestamps).

The cell shows what the rule does to 1080p videos of increasing length under a 16,384-token cap at 2 frames per second, with and without a cap of 768 frames.

Python · cell 7

✏️ A 10-minute video under a 16,384-token cap

A 10-minute 1080p video at 2 frames per second is 1,200 frames. With the 768-frame cap of the released utilities, $F = 768$ , which is 384 temporal steps, and $n_{\text{step}} = \lfloor16{,}384/384\rfloor = 42$ tokens per step. The native-resolution rule shrinks $1{,}920\times1{,}080$ by $\beta = \sqrt{2{,}073{,}600/(42\cdot784)} = 7.94$ and rounds down to $224\times112$ . That is an $8\times4$ grid of 32 tokens per step, and $384\cdot32 = 12{,}288$ tokens in all. Without the frame cap, 600 steps leave 27 tokens each. The frames shrink to $168\times84$ , which is $6\times3 = 18$ tokens, and the total is 10,800. Rounding down to whole tokens leaves a quarter to a third of the budget unspent, and the frames are the size of a thumbnail either way.

The example is the whole difficulty of video in one number. A cap that fits one detailed image buys either a few seconds at full detail or minutes at a resolution where no text is legible. The cell shows the same at the extremes. Thirty seconds get frames of $868\times476$ , while an hour without a frame cap gets frames of $56\times28$ , two tokens for each pair of frames. §29.5 turns this into a probability of seeing a short event and shows how choosing frames by their relevance to the question, and re-reading them, beats uniform sampling. The other levers are the ones of this section. One is fewer tokens per frame by merging or pooling (LLaVA-OneVision reads up to 32 frames at 196 tokens each). The other is longer contexts (Qwen3-VL's native window is 256K tokens).

Core

A video token budget

Write plan_video(duration_s, src_fps, H, W, fps=2.0, max_tokens=16384, max_frames=768, factor=28), the planner of the definition, with temporal patches of 2 frames. The file holds int(duration_s * src_fps) frames, and the frame count never exceeds that, is even, and is at least 2. Return a dict with five entries.

"n_frames", an even int.
"indices", an int array of source frame indices.
"timestamps", a float array of times in seconds.
"size", the resized (h, w).
"tokens", an int.

The setup provides smart_resize(H, W, factor, min_pixels, max_pixels), the native-resolution rule, which returns (h, w). Call it with min_pixels = 4 * factor**2 and max_pixels = b * factor**2, where b is the per-step budget $n_{\text{step}}$ . You may assume the video has at least 2 frames and an aspect ratio below 4.

def plan_video(duration_s, src_fps, H, W, fps=2.0, max_tokens=16384, max_frames=768, factor=28):
    F = int(duration_s * fps)
    idx = np.arange(F) * int(src_fps / fps)
    h, w = smart_resize(H, W, factor, 4 * factor**2, (max_tokens // F) * factor**2)
    return {"n_frames": F, "indices": idx, "timestamps": idx / src_fps, "size": (h, w),
            "tokens": F * (h // factor) * (w // factor)}

Tests, verified solutions, and AI help, open once you add your API key.

Pruning visual tokens inside the language model

Everything so far fixes the token count before the language model runs. A different idea is to let the language model read every visual token for a few layers and then drop the ones it is not using. Chen et al. (2024b) measured how much attention LLaVA-1.5 gives its image tokens, layer by layer. In the first two layers, image tokens receive about half as much attention per token as the system prompt. After layer 2 they receive only 0.21 percent as much. Their method, which they call FastV, prunes accordingly. It runs the first $K$ layers on everything, then keeps only the image tokens that received the most attention.

📐 FastV pruning

Fix a layer $K$ and a ratio $R$ . Run layers $1$ to $K$ on all $n = n_t + n_v$ tokens, $n_t$ of text and $n_v$ of image. Take the attention weights $\mathbf{A}\in\R^{h\times n\times n}$ of layer $K$ , where $h$ is the number of heads and row $i$ holds query $i$ 's weights. Score each image token $j$ by the average attention it receives, $\phi_j = \frac{1}{h}\sum_{\text{heads}}\frac{1}{n - 1}\sum_{i\ne j}A_{ij}$ (the paper's definition). Drop the $\lfloor Rn_v\rfloor$ image tokens with the lowest scores from the hidden states. Run layers $K + 1$ to $L$ on the remaining $\hat n = n_t + n_v - \lfloor Rn_v\rfloor$ tokens, in their original order. Text tokens are never dropped.

The released code differs in a way worth noticing. It scores image tokens by the attention they receive from the last token of the prompt only, averaged over heads. It also gives the kept tokens their original position ids, so RoPE still sees the true distance between, say, the instruction and an image token that survived.

How much does this save? The first $K$ layers run on all $n$ tokens and the remaining $L - K$ layers on $\hat n$ , so the saving is one minus the ratio of the two costs.

∂ The 13B example, step by step

The first $K$ layers run on $n$ tokens and the remaining $L - K$ on $\hat n$ . So the pruned model costs $Kc(n) + (L-K)c(\hat n)$ against $Lc(n)$ for the full one, and the fraction saved is one minus their ratio. For the example, $n = 640$ and $\hat n = 64 + 288 = 352$ . Then $c(640) = 4\cdot640\cdot5{,}120^2 + 2\cdot640^2\cdot5{,}120 + 2\cdot640\cdot5{,}120\cdot13{,}824 = 1.619\times10^{11}$ , and $c(352) = 8.80\times10^{10}$ . The ratio is $(2\cdot1.619 + 38\cdot0.880)/(40\cdot1.619) = 0.566$ , and the saving is 43.4 percent. $\blacksquare$

The paper's Table 1 reports this setting at 55 percent of the baseline FLOPs (84.6 against 154.6 in its units), close to the 56.6 percent computed here. The average over its four benchmarks stays at 73.6. They are two captioning sets (NoCaps and Flickr30k), a visual question set (A-OKVQA), and MMMU. Because the MLP and the projections dominate at these lengths, the saving is nearly the fraction of tokens removed after layer $K$ . Here that is $1 - 352/640 = 45$ percent of the tokens in 38 of 40 layers.

Python · cell 8

With more image tokens, the saving grows past the fraction of tokens dropped, because the quadratic attention term matters more. A 75 percent prune after layer 2 of an 11,000-token page saves 77 percent of the prefill. Pruning later, after layer 8, saves less but leaves the model more layers to decide what matters.

Research

Prune visual tokens after layer K

Write fastv_prune(Hs, A, img_start, img_end, R, criterion="mean"). Hs has shape (n, d), the hidden states after layer $K$ . A has shape (h, n, n), layer $K$ 's attention weights, where row $i$ is query $i$ . The image tokens are positions img_start to img_end - 1, so there are n_v = img_end - img_start of them.

Score the image tokens in one of two ways.

With criterion="mean", use the definition, the average over heads and over all other query rows $i \ne j$ of $A_{ij}$ .
With criterion="last", use the released code's rule, the average over heads of row $n - 1$ .

Drop the int(R * n_v) lowest-scoring image tokens, breaking ties in favor of the earlier position, and keep every text token. Return (H_kept, pos_kept), the kept hidden states in their original order and their original positions as an int array.

import numpy as np

def fastv_prune(Hs, A, img_start, img_end, R, criterion="mean"):
    n = Hs.shape[0]
    score = A.mean(0).mean(1)                                # average attention each token gives
    keep = np.argsort(-score)[: n - int(R * (img_end - img_start))]
    return Hs[keep], np.arange(len(keep))

Tests, verified solutions, and AI help, open once you add your API key.

What attention-based pruning keeps

The released rule has a consequence that is easy to miss. The image is flattened in raster order, row by row from the top, and the question comes after it. So the bottom row of the image sits right before the text, and the top row is about $n_v$ positions further back. If the last token's attention falls off with distance, an image token's score is partly a statement about where it sits rather than what it shows.

The widget models that score as content relevance times a decay $e^{-\alpha\Delta/(n_v - 1)}$ , where $\Delta$ is the token's raster distance from the bottom-right corner and $\alpha$ is the decay strength. It keeps the top $1 - R$ fraction of tokens by that score.

At the default setting only 6 of the object's 40 tokens survive, and 96 percent of the kept tokens lie in the bottom half. With the decay removed, all 40 survive and the kept tokens split roughly evenly between the halves (47 percent in the bottom). A uniform sample is immune to position but blind to content, so it keeps 10 of the 40, about its share $1 - R$ . Spending half the budget on each, as the last option does, keeps 37 of the 40. The decay in the widget is an assumption chosen to show the mechanism, but the pile-up at the bottom is what was measured.

⚠️ Attention scores are not importance scores

Endo et al. (2024) looked at which tokens FastV-style pruning keeps when it prunes early. They found the kept tokens concentrated at the bottom of the image, with most tokens toward the top pruned away. The scores come from the attention of the last text tokens, and the authors traced the bias to RoPE. As §5.5 showed, RoPE's scores for aligned content fall with distance (a ceiling on the score rather than a law). At early layers that is enough for the image rows nearest the text in raster order to win regardless of content. Many benchmarks barely noticed, because their questions can be answered from a few tokens or from language priors, while localization collapsed. Their fix scores without RoPE, prunes in stages, and keeps a uniform sample for coverage. It improved localization benchmarks more than fivefold at comparable savings. The general lesson is to check what a pruning rule keeps on tasks that need the whole image, and not to trust an average score that hardly moves.

One more saving needs no new idea. An image that appears in many requests, such as a shared document or the screenshots of a long agent session, can have its encoder output and its KV cache reused across them. That is the prefix caching of §20.6.

Images as compressed text

The token arithmetic of this section can also be run backwards. If a page of 1,000 words can be read back from 100 visual tokens, then rendering old context as an image is a way to compress it. DeepSeek-OCR (Wei et al. 2025) tested this directly. The measure is the compression ratio, the number of text tokens on a page divided by the number of vision tokens used to read it.

📌 Optical compression (DeepSeek-OCR)

Its encoder has about 380M parameters. An 80M SAM-base, the image encoder of the Segment Anything model, reads 16-pixel patches with window attention. It feeds a 16-times convolutional compressor (two $3\times3$ convolutions with stride 2), which feeds a 300M CLIP-large with global attention. So a $1{,}024\times1{,}024$ page is 4,096 patches but only 256 tokens. Modes range from 64 tokens ( $512\times512$ ) to 400 ( $1{,}280\times1{,}280$ ), plus tiled modes. The decoder is a 3B mixture of experts with 570M active parameters.

On Fox, a benchmark of pages to transcribe, decoding precision is about 97 percent when the compression ratio is below 10, and about 60 percent at 20. For pages of 1,200 to 1,300 text tokens (only four pages) it is 59.1 percent with 64 vision tokens and 87.1 percent with 100. OmniDocBench scores whole-document parsing. There DeepSeek-OCR beats GOT-OCR2.0, an OCR model that uses 256 tokens per page, with 100 tokens. It beats MinerU2.0, a document-parsing system that uses more than 6,000 on average, with fewer than 800.

The result is an empirical finding about one model on two benchmarks, not a law. But it reframes the section. A visual token can carry about ten text tokens' worth of information when the encoder is trained for dense text. So counting tokens is not the same as counting information, and the right number of tokens per image depends on what the image contains.

In PyTorch

The block checks three claims of the section against PyTorch's own operators. Our space-to-depth order differs from F.pixel_unshuffle by a fixed permutation of channels. The shuffle-then-linear connector equals F.conv2d with stride $\kappa$ . And attention over a packed row with a block-diagonal mask, through F.scaled_dot_product_attention, equals attention over each image separately.

🔥 In PyTorchfor a machine with PyTorch

In a real model the packed row goes to a variable-length kernel with cu_seqlens instead of a mask, which skips the masked tiles entirely. With Hugging Face, the native-resolution limits are arguments of the processor, and the image's grid comes back with the inputs, so you can read off the token count before running the model.

Choosing a design

design	tokens for a $1{,}920\times1{,}080$ screenshot	encoder	strengths	weaknesses	used by
resize to the encoder's square	576	pretrained, unchanged	cheapest, simplest	small text and detail destroyed, aspect ratio distorted or padded	LLaVA-1.5, early VLMs
tiles chosen by pixels kept, padded, base image, newline tokens	1,948	pretrained, unchanged	keeps aspect ratio, reuses CLIP	few grids, padding, seams, tokens grow fast	LLaVA-NeXT, LLaVA-OneVision
tiles chosen by aspect ratio, stretched, thumbnail, pixel shuffle	2,304	pretrained, fine-tuned on tiles	up to 40 tiles, 4 times fewer tokens per tile	stretching, seams, tile layout must be tagged	InternVL, NVLM
native resolution, packed, 2D RoPE, $2\times2$ merge	2,691 plus 2	retrained for any size	no seams or padding, any aspect ratio, cost follows content	quadratic encoder cost, needs window attention or caps	Qwen2-VL to Qwen3-VL, Pixtral, Kimi-VL
fixed encoder with pooling, crops at inference	256 per crop	pretrained at high resolution	constant cost per crop	pooling discards detail within each block	Gemma 3
merge similar tokens by content	the base count minus $r$ per encoder block (ToMe), or adaptive (PruMerge keeps 5.5 percent on average)	unchanged, merging inside or at the end of the encoder	adapts to content, training-free	merged tokens have no single grid position, which breaks pixel shuffle, newline tokens, M-RoPE, and grounding	ToMe in ViTs, LLaVA-PruMerge
prune inside the language model	any of the above, then fewer after layer $K$	unchanged	training-free, large prefill savings	attention scores biased by position, hurts localization	FastV and successors

There is no single winner, and frontier models combine several rows. Native-resolution encoders have become common in new open models since 2024 (Qwen2-VL to Qwen3-VL, Pixtral, Kimi-VL). Once the encoder is being retrained anyway, packing and 2D RoPE remove the tiling heuristics. Tiling survives where a strong fixed encoder must be reused.

Where it breaks

Seams and layout. Tiles split words and objects at their borders, and without tags or newline tokens the language model can confuse a $2\times3$ grid with a $3\times2$ one. Overlap, tags, and thumbnails reduce this but do not remove it.

Token explosion. A 40-page PDF at 2,700 tokens per page is more than 100,000 tokens, and a video at useful resolution is millions. Every lever of this section trades detail for count, and long documents still need retrieval or page selection on top.

Uniform sampling misses events. A video budget spread evenly over time sees a three-second event with a probability that grows with the frames per second it can afford (the coverage theorem of §29.5).

Pruning drops what matters. Attention-based pruning is biased by position and keeps the tokens the instruction attends to at that layer, which can exclude the one small line of text the answer needs.

Positions extrapolate poorly. Images larger than those seen in training, or videos longer than the training length, put position ids or time offsets outside the trained range. The small ids of M-RoPE help, absolute time hurts on long videos, and text timestamps are the current answer.

Resizing artifacts. Upscaling very small images to meet a minimum pixel count can push them out of the training distribution. Qwen2-VL reports that too high a minimum hurt OCRBench, which has many tiny images, and MM1 saw its average fall at its largest resolution.

🧭 What to remember

Resolution is the biggest single factor in reading documents, charts, and small text, more than encoder size or connector design, because detail lost in resizing cannot be recovered. Reasoning on clear images gains much less.
Tiling reuses a fixed encoder. One rule picks the grid that keeps the most pixels, pads, and adds a base image and newline tokens (LLaVA-NeXT). The other picks the closest aspect ratio, stretches, and adds a thumbnail (InternVL). Thumbnails, tile tags, and overlapping crops each have measured gains.
Pixel shuffle stacks each $\kappa\times\kappa$ block into one token. It is a lossless rearrangement whose linear projection is a stride- $\kappa$ convolution, and average pooling is its rank- $c$ special case. Merging by content (ToMe's bipartite soft matching) adapts to what the image shows, at the price of tokens that no longer sit at one grid position.
Native-resolution encoders resize to whole tokens, use 2D RoPE, and pack images of any shape into rows with block-diagonal masks. A variable-length kernel then pays only $\sum_in_i^2$ for attention.
M-RoPE gives text, images, and video one position scheme in which an image advances positions by its larger side rather than its token count. Time ids have moved from step counts to absolute time to text timestamps.
A video is a frame and token budget, and under a fixed cap longer videos are read at thumbnail resolution. That is why frame selection, pruning (with care about what attention-based rules keep), and compression all matter.
All of these designs bolt a separately trained encoder onto a language model. §31.6 asks whether one Transformer should instead read and write image tokens from the first step of pretraining, and how to measure what any of these models actually see.

Wang et al. (2024a). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. Native dynamic resolution, M-RoPE, two-frame video patches, and the token-count ablation in Table 7.
Dehghani et al. (2023). Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. Packing, masked attention and pooling, factorized and fractional positions, and why mixed-resolution training is cheaper.
Chen et al. (2024a). How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites. InternVL 1.5, with closest-aspect-ratio tiling, thumbnails, and pixel shuffle to 256 tokens per tile.
Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer. AnyRes at scale, with one token budget shared by single images, multiple images, and video.
Dai et al. (2024). NVLM: Open Frontier-Class Multimodal LLMs. The tile-tag ablation and a careful comparison of decoder-only, cross-attention, and hybrid designs at high resolution.
Bai et al. (2025a). Qwen2.5-VL Technical Report. Window attention in a native-resolution encoder and M-RoPE aligned to absolute time.
Bolya et al. (2023). Token Merging: Your ViT But Faster. Bipartite soft matching, proportional attention, and the speed and accuracy of merging without retraining. Shang et al.'s LLaVA-PruMerge carries the idea into a vision-language model.
Chen et al. (2024b). An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. FastV, the attention statistics behind it, and its FLOP formula. Read it together with Endo et al.'s Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration.
Wei et al. (2025). DeepSeek-OCR: Contexts Optical Compression. How many text tokens one visual token can carry, measured.

Made for larger screens

High resolution, documents, and video

Resolution decides what the model can read

What more pixels cost

Tiling an image for a fixed-size encoder

AnyRes grid selection and token count

Choosing tiles by aspect ratio

Thumbnails, tile tags, and seams

Merging neighbors with pixel shuffle

Pixel shuffle and its inverse

Merging by content instead of by position

Reading images at their own size

Pack images of any shape

Native-resolution encoders in practice

Positions in the language model

M-RoPE position ids for a conversation

Video as a frame and token budget

A video token budget

Pruning visual tokens inside the language model

Prune visual tokens after layer K

What attention-based pruning keeps

Images as compressed text

In PyTorch

Choosing a design

Where it breaks

Read more

Watch next