How a photograph becomes a sequence of patch tokens, why a model with almost no built-in knowledge of images beats the convolutional networks that had it, and how the same tokens are trained, made hierarchical, and turned into boxes and masks. From the first strided convolution to SAM 3, at interview depth.

This chapter is about the Vision Transformer, or ViT, a model that reads an image the way a language model reads a sentence. It cuts the image into small squares, turns each square into a vector called a token, and runs an ordinary Transformer over the sequence. That one design now sits inside most of vision. The image encoders of CLIP and of vision-language models are ViTs, and so are the encoder of Segment Anything and the diffusion Transformers that generate images.

Two earlier ideas carry the chapter. A Transformer is a stack of layers that updates a sequence of vectors through attention, which compares every token with every other token (Chapter 5). A convolutional network, or CNN, slides small filters over the image, so each layer looks only at a small neighborhood and applies the same weights at every position (§0.8). Those two assumptions, locality and weight sharing, are much of why a CNN learns from fewer images. A ViT drops locality, since any patch can attend to any other from the first layer. It keeps weight sharing only in a weak form, since every patch passes through the same weights but also carries its own learned position. §27.3 measures what that trade costs and when it pays.

This section covers the first step, turning pixels into tokens. Nearly every detail of it is a standard interview question, from the order of the pixels inside a patch to what happens when the image changes size. The obvious way fails. Give a Transformer one token per pixel, and a $224\times224$ photograph becomes $50{,}176$ tokens. Each head of each layer then computes $50{,}176^2 \approx 2.5\times10^9$ scores, one for every pair of tokens. Stored as 16-bit numbers, that is about 5 GB for one head of one layer.

The ViT of Dosovitskiy et al. (2021) fixes this with one decision. It cuts the image into $16\times16$ squares, called patches, and makes each patch one token. A $224\times224$ image then has $14\times14 = 196$ tokens and $196^2 = 38{,}416$ scores per head, which is $65{,}536$ times fewer. A completely standard Transformer can read that. §1.8 sketched this step, and its Exercise 1.8a cut an image into patches with one reshape and one transpose.

Filling in the details means answering five questions. In what exact sense is the patch embedding "the same as a convolution"? What does the patch size buy, and what does it cost? Can you change the patch size after training? How does a set of tokens become one vector that describes the image? And since attention cannot see order, how does the model know where each patch came from, and what happens to that knowledge when the image changes size?

How Vision Transformers are named

A ViT's name gives its size and its patch side. ViT-B/16 is the Base size with $16\times16$ patches, and ViT-L/14 is the Large size with $14\times14$ patches. When a resolution is added, as in "ViT-L/14 at 336" or "ViT-L/14@336px", it is the side of the square input image in pixels. The Base, Large, and Huge sizes come from the ViT paper, which took the first two from the BERT text encoder. The Tiny and Small sizes come from DeiT, a recipe for training ViTs on ImageNet alone (Touvron et al. 2021).

Here $L$ is the number of Transformer blocks, $d$ the width of every token, $h$ the number of attention heads, and $d_{\text{ff}}$ the width of the hidden layer of each block's two-layer network (its MLP). §27.2 counts the parameters of every size, along with the giant sizes g and G and the 22-billion-parameter ViT-22B.

Why pixels cannot be tokens

A Transformer layer does two kinds of work on $T$ tokens, each a vector of $d$ numbers. The linear layers, meaning the attention projections and the MLP, transform each token on its own, so their work grows like $Td^2$. Attention compares every pair of tokens, so its work grows like $T^2d$. Plain attention also stores the scores during training, and that memory grows like $hT^2$ for $h$ heads (§4.7). With one token per pixel, $T$ is the number of pixels, and the $T^2$ term is fatal at any useful resolution.

The field tried three ways out. The first shrinks the image until the pixels are few. Image GPT (Chen et al. 2020) made each pixel a token, but only on images of at most $64\times64$ pixels, with the colors clustered into a palette of 512. The second keeps every pixel but lets a small, fixed set of learned vectors read them through cross-attention, in which the queries come from one set and the keys and values from another. The cost then grows with the number of pixels times the number of learned vectors, not with the square of the pixel count. This is the design of the Perceiver (Jaegle et al. 2021), which §4.6 covers. The third groups pixels into patches before the Transformer sees anything.

Patches won. They keep every pixel value in the input, cost one matrix multiply, and leave an ordinary Transformer behind them.

A patch is a vector the model projects

In a language model a token is an integer, and the model turns it into a vector by reading one row of an embedding table (§1.2). A patch has no integer. It arrives as a vector already, the $P\cdot P\cdot C$ raw numbers of a $P\times P$ square with $C$ channels. A color image has three channels, one each for red, green, and blue. So the table lookup becomes a matrix multiply. Every patch is multiplied by the same learned matrix. Each column of that matrix is a pattern detector, and each number of the resulting token is the dot product of the patch with one detector.

A tiny case with four pixels per patch and two detectors shows everything that happens.

That last patch shows, in miniature, what a patch embedding can lose. The null space of the map $\mathbf{x}\mapsto\mathbf{x}^\top\mathbf{W}_E$ is the set of patches it sends to zero. This map takes four numbers to two, so its null space is two-dimensional. It contains the vertical edge $(1, -1, 1, -1)$, the horizontal edge $(1, 1, -1, -1)$, and every mix of the two. In general, a matrix that maps $P^2C$ numbers to $d < P^2C$ numbers must send some patterns to zero. One with $d \ge P^2C$ can be injective, meaning different patches always get different tokens, and then nothing is lost at this step. We come back to this with real model sizes below.

The patch embedding, stated exactly

Now the general case. An image is $\mathbf{X}_{\text{img}}\in\R^{H\times W\times C}$, with height $H$, width $W$, and $C$ channels. In the math the channels come last. PyTorch stores images as $(C, H, W)$, and the code always says which layout it uses. The patch side is $P$ pixels, and for now $P$ divides both $H$ and $W$. The patches form a grid of $H_p = H/P$ rows and $W_p = W/P$ columns, so there are

patches. We write $N_p$ rather than the ViT paper's $N$, because $N$ counts parameters in this book and the next section needs both.

In plain words, cut the image into a grid, flatten each square, multiply every square by the same matrix, and add the same bias.

A note on conventions. As everywhere from Chapter 4 on, a weight matrix has shape (input, output) and multiplies token row vectors from the right. §1.8 wrote the same map in column form, $\mathbf{W}_{\text{p}}\mathbf{x}_i$ with $\mathbf{W}_{\text{p}}\in\R^{d\times P^2C}$, so $\mathbf{W}_E = \mathbf{W}_{\text{p}}^\top$. The ViT paper calls this matrix $\mathbf{E}$. We rename it because $\mathbf{E}$ is the token embedding table everywhere else in the book.

For ViT-B/16 ($P = 16$, $C = 3$, $d = 768$) each patch holds $16\cdot16\cdot3 = 768$ numbers, and the patch embedding has $768\cdot768 + 768 = 590{,}592$ parameters. That is under 0.7 percent of the model's 86.6 million. The first layer of a ViT is tiny. Almost all of the cost lies in the Transformer behind it.

When the image does not divide

So far $P$ divided both sides. When it does not, there are three options, and libraries differ on which they take.

• Crop. Keep $\lfloor H/P\rfloor$ rows of whole patches and drop the last $H \bmod P$ rows of pixels. The strided convolution that most libraries use for the patch embedding (the next subsection) does exactly this, without a warning. Its output has $\lfloor (H - P)/P\rfloor + 1 = \lfloor H/P\rfloor$ rows.
• Pad. Pad the image up to the next multiple of $P$. That gives $\lceil H/P\rceil$ rows, and the last patches are partly empty. timm, the PyTorch Image Models library that most vision code builds on, does this when built with dynamic_img_pad=True.
• Resize. Resample the image to a multiple of $P$ before cutting it into patches. This changes the apparent size of objects.

By default, timm's patch embedding accepts only the image size the model was built for and raises an error otherwise. That is the safest behavior of all.

The Transformer also reads a few learned tokens that are not patches, placed in front of them and called prefix tokens. Most ViTs use a class token, a learned vector whose final value summarizes the image (defined later in this section). Some also add register tokens, extra learned slots that §27.4 explains. With $n_{\text{pre}}$ prefix tokens, the sequence length is $T = N_p + n_{\text{pre}}$. For ViT-B/16 at 224 with one class token, $T = 196 + 1 = 197$.

def patch_token_count(H, W, P, n_prefix=1, mode="crop"):
    raise NotImplementedError

Three ways to compute the same map

The definition can be computed three ways, and production code uses all of them.

1. Reshape, then multiply. Cut the image into patches with a reshape and a transpose, as in Exercise 1.8a, then compute $\mathbf{X}_p\mathbf{W}_E$ with one matrix multiply.
2. A linear layer per patch. Apply nn.Linear(P*P*C, d) to every patch vector. This is the same arithmetic. PyTorch stores the weight as $\mathbf{W}_E^\top$, with shape $(d, P^2C)$.
3. A strided convolution. Apply nn.Conv2d(C, d, kernel_size=P, stride=P) to the image and read the output grid in raster order. The original ViT code, timm, and most libraries implement it this way.

The third surprises people, so here it is exactly. A convolution slides a small grid of weights, the kernel, across the image. At each stop it multiplies the kernel with the pixels under it and adds up the products. That gives one output number per filter, where a filter is one kernel with its own weights. The stride is how many pixels the kernel moves between stops. §0.8 builds convolutions, strides, and output sizes from scratch. In symbols, take a convolution with $d$ filters $\mathbf{W}_{\text{conv}}\in\R^{d\times C\times P\times P}$ (PyTorch's layout), stride $s$, and no padding. For output channel $o$ at output position $(r, c)$ it computes

and the output has $\lfloor (H - P)/s\rfloor + 1$ rows. With $s = P$ that is $H/P = H_p$ rows, and likewise $W_p$ columns.

In plain words, a convolution applies one small dense layer to every window it visits. When the stride equals the kernel size, the windows tile the image with no overlaps and no gaps, so each pixel is used exactly once. Then "a dense layer on every window" is exactly "a dense layer on every patch".

This also explains why the patch embedding is cheap. Fast convolution code often uses im2col, short for "image to columns". It copies every window the kernel visits into a row of a matrix and then does one matrix multiply (§0.8). For a $k\times k$ kernel at stride 1, the copies take about $k^2$ times the memory of the input. With kernel equal to stride, the windows do not overlap, so im2col copies nothing. It is a reshape and a transpose, the same patch cutting as in Exercise 1.8a.

The one trap is the order in which a patch is flattened. Every wrong order still produces tensors of the right shape, so nothing crashes. The definition above flattens channels last, so pixel $(u, v)$ in channel $c'$ goes to index $(uP + v)C + c'$. PyTorch's Conv2d weight and F.unfold flatten channels first, to index $c'P^2 + uP + v$.

Here is one pixel in both orders. Take $P = 16$, $C = 3$, and pixel $(u, v) = (2, 5)$ in channel 1. It sits at index $(2\cdot16 + 5)\cdot3 + 1 = 112$ in the channels-last order and at index $1\cdot256 + 2\cdot16 + 5 = 293$ in the channels-first order. In NumPy, the matrix for channels-last patches is W_conv.transpose(2, 3, 1, 0).reshape(P*P*C, d). The matrix for channels-first patches is W_conv.reshape(d, C*P*P).T. An interviewer who hands you a checkpoint is testing exactly this.

The cell checks the hand example. Then it checks the theorem on a random color image, computing the convolution the slow way, window by window.

Both forms give the four tokens of the hand example, $(2, 2)$, $(8, 0)$, $(12, 0)$, and $(2, -2)$. On the random image they agree to about $10^{-14}$, which is rounding error in 64-bit floats.

import numpy as np

def conv_weight_to_linear(W_conv):
    raise NotImplementedError

def patch_embed_conv(img_chw, W_conv, b):
    raise NotImplementedError

Overlapping patches and convolutional stems

Kernel equal to stride is a choice, not a law. A convolution whose kernel is larger than its stride gives overlapping patches. Neighboring tokens then share pixels, so an edge that falls on a patch border is seen whole by some token. Xiao et al. (2021) went further. They replaced ViT's single $16\times16$ stride-16 convolution with a few stacked $3\times3$ stride-2 convolutions, called a convolutional stem, that reach the same total stride of 16. The token count is unchanged, and they removed one Transformer block so that the total compute matches. Yet training became much less sensitive to the optimizer and its settings, and ImageNet-1k accuracy rose by about 1 to 2 points.

These designs put back some of the machinery of convolutional networks, whose built-in assumptions §27.3 examines. The ViT paper itself tried a hybrid, in which each "patch" is one $1\times1$ cell of the feature map of a convolutional network (a ResNet). The hierarchical backbones of §27.5 start from small $4\times4$ patches. That way they can produce the fine feature maps, one vector per 4 pixels, that detection and segmentation need.

In an interview, the patch embedding is the place to say that a ViT is not free of image-specific assumptions. It assumes that $P\times P$ squares are meaningful units.

What the patch size trades

The patch side is the most consequential choice in a ViT. Everything the model costs follows from its token count, and the token count follows from $P$.

The cell sweeps the patch sizes in use today at two resolutions, for a model of width $d = 768$ with one class token. "Pixels per token" is $P^2$. The last column is the attention-score work relative to ViT-B/16 at 224.

Three things stand out. First, the attention work spans a factor of about 34,000 across sizes that all appear in real models. It runs from 2,500 scores per head at $P = 32$ and 224 pixels to 85 million at $P = 4$ and 384 pixels. Second, the patch embedding gets smaller as patches shrink, since it has $P^2Cd + d$ parameters. The extra cost of small patches is all in the Transformer that reads them. Third, $P = 14$ does not divide 384, since $384/14 \approx 27.4$. By the crop rule above, a convolution reads only the top-left $378\times378$ pixels and silently drops the rest.

The widget shows the same trade on a picture. Each patch on the right is replaced by its average color. That is the crudest summary a token could carry, and it shows how much detail must survive inside one token.

What is lost inside a patch

Go back to the null space of the hand example. The rank of a matrix is the number of independent directions it can output. A patch embedding $\mathbf{W}_E\in\R^{P^2C\times d}$ has rank at most $\min(P^2C, d)$. So when $d < P^2C$, at least $P^2C - d$ directions of pixel space map to zero. They are gone before the first attention layer. When $d \ge P^2C$, a learned $\mathbf{W}_E$ can be injective, and nothing is lost at this step.

The standard models sit on both sides of that line.

• ViT-B/16 maps $16\cdot16\cdot3 = 768$ numbers to $d = 768$. The matrix is square and can be invertible.
• ViT-L/16 maps 768 numbers to 1,024, and ViT-H/14 maps $14\cdot14\cdot3 = 588$ numbers to 1,280. Both have room to spare.
• ViT-B/32 maps 3,072 numbers to 768, so at least 2,304 directions of each patch are discarded. ViT-S/16 maps 768 numbers to 384.

What does a model short of dimensions keep? The ViT paper plotted the top principal components of ViT-L/32's learned first-layer filters, the directions that carry most of their variation (their Figure 7, left). They are smooth patterns rather than noise, plausible building blocks for the fine structure inside a patch. Smooth patterns are what natural image patches mostly contain, so they are the right directions to keep.

The larger loss is not the projection. It is what the model can afford to compute on. Each token must summarize $P^2$ pixels, and attention mixes information only between whole tokens. So detail finer than a patch must survive inside the $d$ numbers of a single token. Wang et al. (2025) measured what happens when you stop compressing.

In practice this is the main design question about patches. An interviewer who asks "why not use smaller patches?" wants three things. Smaller patches help. They cost about $16\times$ in attention per halving. And the choice is set by the compute you have.

Changing the patch size of a trained model

A ViT trained with $P = 16$ cannot be run with $P = 8$ by changing a number. Two parts of the model depend on $P$, and nothing else does. The patch embedding is a bank of $16\times16$ filters, and the position table has one row per patch of a $14\times14$ grid. Beyer et al. (2023) asked how to resize the patch-embedding filters so that the resized model behaves like the original. Their answer, part of a model they call FlexiViT, is short enough to prove here.

Think of each column of $\mathbf{W}_E$ as a filter $\boldsymbol\omega$, a small image of weights the same size as a patch. The token is the dot product $\langle\mathbf{x}, \boldsymbol\omega\rangle$ of the patch $\mathbf{x}$ with the filter. Suppose you want to run the model on an image twice as large, with patches twice as large, so that each patch still covers the same part of the scene. The obvious idea is to resize the filters like images. It fails, and the smallest case shows why.

The general case works the same way. Resizing a patch with bilinear interpolation is a linear map. So for single-channel $p\times p$ patches there is a matrix $\mathbf{B}\in\R^{p_*^2\times p^2}$ with $\operatorname{vec}(\operatorname{resize}(\mathbf{x})) = \mathbf{B}\operatorname{vec}(\mathbf{x})$, where $p_*$ is the new side. A color patch resizes each channel separately with the same $\mathbf{B}$. In the example, $\mathbf{B}$ is the column $(1, 1, 1, 1)^\top$.

If you resize both the patch and the filter, the token becomes $\langle\mathbf{B}\mathbf{x}, \mathbf{B}\boldsymbol\omega\rangle$. Each original pixel now covers about $(p_*/p)^2$ new pixels, and each of them adds its product with the filter to the token. So the token grows by about $(p_*/p)^2$, and doubling the patch side quadruples it. What we want instead is a filter $\hat{\boldsymbol\omega}$ that gives the same token on the resized patch, $\langle\mathbf{B}\mathbf{x}, \hat{\boldsymbol\omega}\rangle = \langle\mathbf{x}, \boldsymbol\omega\rangle$, for every patch $\mathbf{x}$.

The tool is the Moore-Penrose pseudoinverse $\mathbf{A}^+$ of a matrix $\mathbf{A}$. It gives the least-squares solution of $\mathbf{A}\mathbf{y} \approx \mathbf{t}$ as $\mathbf{y} = \mathbf{A}^+\mathbf{t}$. Among all least-squares solutions it picks the shortest (§0.10). FlexiViT's rule applies it to $\mathbf{B}^\top$, and the paper calls it PI-resize, for pseudoinverse resize.

In the one-pixel example, $\mathbf{B}^\top\mathbf{B} = 4$ and $(\mathbf{B}^\top)^+\omega = \mathbf{B}(\mathbf{B}^\top\mathbf{B})^{-1}\omega = (\omega/4, \omega/4, \omega/4, \omega/4)$, the filter we found by hand.

In plain words, when the new patch has at least as many pixels as the old one, some filter undoes the resize exactly, and the pseudoinverse finds it. When it has fewer, some information is gone, and the pseudoinverse gives the best linear compromise for patches that look like random noise.

The cell builds the bilinear resize matrix with half-pixel centers. Each pixel is treated as the center of a small square cell, and the resized grid covers the same squares. PyTorch calls this align_corners=False, and the last part of this section defines it precisely. The cell resizes filters up and down and measures the token error on random smooth patches.

The patches and filters here are smooth, built by upsampling random $4\times4$ grids. Natural image patches and learned filters are also dominated by slowly varying patterns. The cell shows four things.

• Naive resizing scales the tokens by close to $(p_*/p)^2$, by 3.77 for $8\to16$ against the predicted 4.
• Dividing by $(p_*/p)^2$ fixes the scale but still leaves errors of about 3 to 9 percent.
• PI-resize reproduces the original tokens to rounding error whenever $p_* \ge p$.
• Going down from 16 to 12 pixels, no filter can be exact for every patch. Yet on smooth patches PI-resize misses by only about 0.1 percent, since almost nothing they contain is lost in the resize.

One special case is worth knowing. Downsampling by exactly 2 with half-pixel bilinear resizing averages each $2\times2$ block, so $\mathbf{B}\mathbf{B}^\top = \frac14\mathbf{I}$. PI-resize then reduces to resizing the filter and multiplying it by 4.

FlexiViT then trains one model for many patch sizes at once. It stores the patch embedding as $32\times32$ filters and the positions as a $7\times7$ table. At every training step it draws a patch size uniformly from $\{48, 40, 30, 24, 20, 16, 15, 12, 10, 8\}$, every divisor of its 240-pixel images in that range. It then resizes both on the fly, the filters with PI-resize and the positions bilinearly. At deployment the trained model can trade accuracy for speed by choosing its patch size, with no retraining.

import numpy as np

def pi_resize(w, B):
    return B @ w          # placeholder: plain bilinear resizing of the filters

From many tokens to one vector

A Transformer returns one output vector per input token. After $L$ Transformer blocks (§27.2 builds them), the model holds $T$ output vectors, the rows of $\mathbf{X}^{(L)}\in\R^{T\times d}$. But a classifier, a retrieval index, or a contrastive loss needs one vector $\mathbf{z}\in\R^d$ per image. Turning many vectors into one is called pooling. Three ways of doing it are in use, and they are worth knowing precisely, because codebases mix them up.

The first comes from text. The BERT encoder puts a special [CLS] token in front of the sentence and reads a summary of the whole sentence off its final vector (§7.1). ViT copies this with a learned class token $\mathbf{x}_{\text{cls}}\in\R^d$ placed in front of the patch tokens. Attention lets this token read every patch in every layer. It starts as the same vector for every image, so its final value is whatever summary of the patches the loss rewards.

The second is the habit of convolutional networks. It averages the outputs at every position and is called global average pooling (GAP). The third, multihead attention pooling (MAP), lets a learned query vector decide how much each patch contributes. It is the pooling with one learned query of §4.6, a single query that cross-attends to the set, here with several heads.

Rows of $\mathbf{X}^{(L)}$ are numbered from 0. Row 0 is the class token if there is one, the next $R$ rows are any register tokens, and the patch tokens follow. So patch $i$ sits in row $n_{\text{pre}} + i$, where $n_{\text{pre}} = 1 + R$ counts the prefix tokens.

The two orders of GAP are not a detail. Two codebases average first and normalize after. One is timm. The other is the fine-tuning code of the masked autoencoder (MAE) of §28.5, a ViT pretrained by filling in hidden patches. Google's big_vision codebase, which trained many of the large ViTs, does the opposite. It applies its final LayerNorm to every token and then averages. A checkpoint trained one way and evaluated the other way gets the wrong features, and nothing crashes.

Mean pooling has one property the class token lacks. It does not depend on the order of the patches, since a sum does not care about order. Attention pooling with a learned query has the same property, because cross-attention gives the same output however its keys and values are ordered (§4.6). The cell checks the first claim and measures how far apart the two GAP orders are.

The tokens in this cell share an offset, as real final-layer tokens do. Four of them have a very large norm, like the artifact tokens that §27.4 studies. The two orders disagree in direction, with a cosine of about 0.95, and in length, 8.00 against 7.09. Averaging first lets the four large tokens pull the mean toward themselves. Normalizing first caps each token's influence.

The lengths differ for a simple reason. With gain 1 and shift 0, a normalized vector has mean 0 and variance 1 over its $d$ entries, so its norm is $\sqrt d = 8$, up to the $\epsilon = 10^{-6}$ that the cell adds to the variance. The mean-then-normalize vector is one normalized vector, so its norm is 8. The normalize-then-mean vector averages 196 vectors of norm 8 that point in different directions, which makes it shorter. A linear head trained on one sees the wrong scale and the wrong direction on the other.

Which head is best was settled by careful experiments, not by argument.

In an interview, "class token or mean pooling?" has a precise answer. They perform about the same once each has its own learning rate. Mean pooling avoids an extra token. And the LayerNorm order must match the checkpoint.

import numpy as np
from scipy.special import erf

def map_head(X, n_prefix, params):
    raise NotImplementedError

Telling the model where each patch came from

Self-attention cannot see order. §4.6 proved that an encoder with no position information is permutation equivariant. If you shuffle its input tokens, its output tokens shuffle the same way, and a mean-pooled vector is unchanged. So a ViT without positions reads a bag of patches, a collection with no arrangement. It can still tell a cat's fur from a dog's. But it cannot tell an eye above a mouth from an eye below one.

So we add to each token a vector that says where it is. For the patch at grid row $r$ and column $c$, write this vector $\mathbf{p}_{(r,c)}\in\R^d$. Stacked over the whole sequence, with rows for the prefix tokens too, these vectors form the position table $\mathbf{P}_{\text{pos}}\in\R^{T\times d}$. It is the position table $\mathbf{P}$ of §5.4 with a subscript, since italic $P$ is the patch size here.

How much do positions matter for images? The ViT paper measured it with a cheap test that it uses for quick comparisons.

The test measures the features alone, since nothing in the network is trained. It is far cheaper than fine-tuning, which is why the paper uses it for ablations, experiments that remove or swap one part to see what it contributes. With only five examples per class its numbers sit well below fine-tuned accuracy, so they are compared only with each other.

Why does a plain 1-D table do as well as a 2-D one? Because it learns the 2-D structure by itself. Figure 7 (center) of the ViT paper takes each patch of ViT-L/32's $7\times7$ grid and shows the cosine similarity of its learned position vector with all the others. Nearby patches have similar vectors. Patches in the same row or the same column are more similar than others. All of this is learned from the raster index and the data.

The paper's own guess is that a $14\times14$ grid is small, so any reasonable scheme learns it equally easily. At higher resolutions, at many resolutions, and at many aspect ratios, the choice starts to matter. The rest of this part covers the three schemes in use.

Learned tables, one-dimensional and factorized

The default ViT learns $\mathbf{P}_{\text{pos}}$ outright, one free vector per token, initialized with small random values. With a class token that is $(N_p + 1)d$ parameters. The ViT paper's "2-D" variant factorizes the table. It learns a row table and a column table, each with vectors of width $d/2$, and concatenates them, $\mathbf{p}_{(r,c)} = [\mathbf{p}^{\text{col}}_c;\ \mathbf{p}^{\text{row}}_r]$. That needs $(H_p + W_p)\,d/2$ parameters instead of $H_pW_pd$. It can also cover a larger grid by extending each table. In the ablation it made no difference. NaViT (Dehghani et al. 2023), which reads images at their native size, factorizes the same way but adds two full-width vectors instead of concatenating them.

Fixed two-dimensional sines and cosines

The second scheme learns nothing. §5.4 built the sinusoidal encoding for text. It gives position $t$ the sines and cosines of $t$ times a ladder of frequencies, from fast to slow. The grid version gives half of the channels to the column index and half to the row index.

Beyer et al. (2022) found it better than a learned table in their ImageNet-1k recipe, 76.5 against 75.0 percent top-1 at 90 epochs. The gap shrinks to 80.0 against 79.6 at 300 epochs. The masked autoencoder, MAE (He et al. 2022), uses it in both its encoder and its decoder.

The layout is easiest to see on one row with very few channels. With $d = 8$ there are only two frequencies, one fast and one slow, so every number can be checked by hand.

The squared length came out as $d/2$ for this row, and that is no accident. The dot product of any two rows of the table has a simple form, and it depends only on how far apart the two patches are.

In plain words, the similarity of two positions is a row term plus a column term, and each depends only on how far apart the patches are. Every patch sees the same similarity pattern, centered on itself. Patches in the same row or column are especially similar. That is the structure learned tables discover on their own. The cell draws the cosine-similarity map for three patches of a $14\times14$ grid at $d = 768$.

The plus-shaped pattern is the row term plus the column term. Notice also how high the similarities are everywhere, never below about 0.71 for patch (5, 9). Most of the 192 frequencies are so slow that they barely change across 14 patches. They add a nearly constant amount to every dot product. The position signal is carried by the few fast frequencies, and that matters when we resize the table below.

import numpy as np

def sincos_2d(H_p, W_p, d, base=10000.0):
    raise NotImplementedError

Rotary positions on a grid

The third scheme moves positions out of the input and into attention. Rotary position embedding, RoPE for short, splits each query and key into pairs of numbers. It rotates each pair by an angle proportional to the token's position (§5.5). The dot product of two rotated pairs depends only on the difference of their angles. So every attention score depends on position only through the offset between the query and the key. The grid version does this once per axis.

In plain words, attention between two patches can depend on what they contain and on how far apart they are vertically and horizontally. It never depends on where they sit in absolute terms. That is the right assumption for an image that might be any size.

The cell checks it. Two pairs of patches with the same offset get the same score, wherever they are. A different offset gives a different score.

The payoff shows up when the resolution changes, and one experiment measures it.

In RoPE-Mixed a pair can rotate by $\theta^x c + \theta^y r$, so it can represent diagonal directions. Heo et al. use base $b = 100$ rather than $10^4$. Their reason is that a grid's side is only the square root of its token count, so they take the square root of the base too. A smaller base raises every frequency, so fewer channels are spent on angles that barely move across a grid only a few dozen patches wide. DINOv3 (Siméoni et al. 2025) uses RoPE with patch coordinates scaled into the box $[-1, 1]$. During training it rescales that box by a random factor in $[0.5, 2]$, so the model learns to handle many resolutions and aspect ratios.

Every implementation has to decide what to do with the prefix tokens. The class token and the registers have no place on the grid. So timm's version of EVA-02, a family of ViTs that use axial RoPE, rotates only the patch tokens, and so does DINOv3. They leave the first $n_{\text{pre}}$ queries and keys as they are. A rotation by angle zero changes nothing, so an unrotated token behaves exactly like a token at grid position $(0, 0)$.

By the theorem, the score between the class token and a patch then depends on where that patch is. Scores between two patches stay purely relative. With DINOv3's coordinates in $[-1, 1]$, that silent origin is the center of the image. And with no prefix tokens at all, shifting every patch by the same offset leaves every score unchanged, and so the whole attention output too.

import numpy as np

def rope_2d_axial(x, rows, cols, n_prefix=0, base=100.0):
    raise NotImplementedError

def rope_2d_attention(Q, K, V, rows, cols, n_prefix=0, base=100.0):
    raise NotImplementedError

The table sums up the five ways to give a ViT positions.

Some models pack images of many sizes and aspect ratios into one batch, each at its native resolution. NaViT and its successors push this furthest, and they are covered with vision-language models at native resolution.

In an interview, the position question is usually "how would you serve a ViT at a resolution it was not trained at?" The answer depends on the scheme, and the next part covers the most common case, a learned table.

New resolution, same model

Fine-tuning at a higher resolution than in pretraining usually improves accuracy (§27.4 explains why). A ViT handles the change gracefully in every part but one. Keep $P$ fixed, and a $384\times384$ image becomes a $24\times24$ grid of 576 patches instead of $14\times14$. The patch embedding, the attention, and the MLPs work on any number of tokens. But a learned position table has exactly $14\cdot14 + 1$ rows.

The ViT paper's fix, in its section on fine-tuning, is to treat the table as a $14\times14$ image with $d$ channels and resize it. It uses 2-D interpolation "according to their location in the original image". The class token's row is not part of the grid and is kept as it is. The paper notes that two steps put knowledge of the 2-D structure of images into the model by hand, this resizing and the cutting into patches. Nothing else does.

Resizing a grid has one subtle convention, and it is the source of many silent bugs. Which old coordinate does each new grid point read from? A tiny case shows the usual answer.

In plain words, each new cell reads the old grid at its own center. With this rule a $14\times14$ table resized to $14\times14$ comes back unchanged, and the grid covers the same image at every size.

What does interpolation do to a table? For the sine-cosine table we can check exactly, since we know the formula at any fractional position. The cell resizes the $14\times14$ table to $24\times24$. It compares the result with the formula evaluated at the very coordinates $s(i')$ that the new grid reads.

The fastest frequency, one radian per patch, comes back with errors of about 0.12. The slow ones are reproduced almost perfectly. The similarity maps of the interpolated and exact tables still have a correlation above 0.999. Linear interpolation is a low-pass filter, meaning it keeps slowly varying patterns and damps fast ones. So it keeps the coarse layout and blurs the finest positional detail, and the finest detail is what tells neighboring patches apart.

In practice there are two cases. For the sine-cosine table you do not need to interpolate at all, since you can recompute it at the new coordinates. For a learned table there is no formula, so interpolation followed by brief fine-tuning is the standard fix. Language models face the same problem when a context grows past its training length. Position interpolation for long contexts is the one-dimensional version of this step.

import numpy as np

def resize_pos_embed(P_pos, n_prefix, old_grid, new_grid):
    raise NotImplementedError

The ViT input sequence

All the pieces now fit together. The class token goes first, then any registers, then the embedded patches in raster order, and the position table is added to the whole sequence.

Registers carry no position because they do not belong anywhere in the image. In the code of DINOv2, and in timm's version of it, they are inserted after the position embedding has been added.

The Transformer that reads $\mathbf{X}^{(0)}$ is the subject of §27.2.

In PyTorch

The block checks the three forms of the patch embedding against each other. They are the convolution, F.unfold with channels-first weights, and the channels-last reshape with the permuted weight. It then assembles $\mathbf{X}^{(0)}$ with a class token, four registers, and 2-D sine-cosine positions. Finally, it confirms that F.interpolate with align_corners=False computes the half-pixel formula.

In practice you rarely write these by hand. timm loads a checkpoint at a new resolution and resizes its positions for you. It also ships PI-resize for patch embeddings, with bicubic resizing and antialiasing by default rather than the bilinear resize of our derivation. The calls below are shown, not run.

Read more

• Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. The Vision Transformer. Read Section 3 for the patch embedding and the resolution change, and Appendix D.3 and D.4 for the pooling and position ablations.
• Beyer, Zhai & Kolesnikov (2022). Better plain ViT baselines for ImageNet-1k. A short note whose Table 1 isolates the effect of GAP and fixed 2-D sine-cosine positions.
• Beyer et al. (2023). FlexiViT: One Model for All Patch Sizes. Section 3.4 and Appendix A.2 derive PI-resize, and the rest shows one model serving every patch size.
• Zhai et al. (2022). Scaling Vision Transformers. Section 3.2 compares the class token with GAP and MAP heads and explains the TPU padding argument.
• Xiao et al. (2021). Early Convolutions Help Transformers See Better. Replacing the patchify stem with a few small convolutions, and why it stabilizes training.
• Heo et al. (2024). Rotary Position Embedding for Vision Transformer. Axial and mixed-frequency 2-D RoPE, with multi-resolution results in Appendix A.2.
• Wang et al. (2025). Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More. What happens to accuracy as patches shrink all the way to single pixels.
• He et al. (2022). Masked Autoencoders Are Scalable Vision Learners. The source of the 2-D sine-cosine convention used here, in its released code's util/pos_embed.py.

Watch next

▶

▶