This section is about the assumptions a model makes before it sees any data, and why a Vision Transformer, which assumes almost nothing about images, can still beat a network built around them. A convolutional network, or CNN, assumes two things about pictures (§0.8). Nearby pixels belong together, so each layer looks only at a small neighborhood. And a pattern means the same thing wherever it appears, so the same small filter is applied at every position. A Vision Transformer, or ViT, cuts the image into square patches, turns each patch into a token, and lets every token attend to every other token from the first layer on (§27.2). It drops the first assumption entirely. It keeps only a weak form of the second, since every patch passes through the same weights but also carries a learned position, so nothing forces a pattern to mean the same thing in every place.

Throwing that knowledge away looks like a mistake, and with a million training images it is. The score here is top-1 accuracy, the share of test images whose highest-scoring class is the right one. A name like ViT-B/16 gives the model's size and patch side, here the Base size (about 86 million parameters) with 16 by 16 pixel patches (§27.1). Trained on ImageNet-1k alone (1.28 million labeled photos in 1,000 classes) and fine-tuned at 384 by 384 pixels, ViT-B/16 reaches 77.9%. ViT-L/16, the Large size with about 3.5 times as many parameters, does worse, 76.5% (Dosovitskiy et al. 2021, Table 5). Two strong CNNs of the time did far better on the same data, 82.9% for RegNetY-16GF at 224 pixels and 84.3% for EfficientNet-B7 at 600 pixels (Touvron et al. 2021, Table 5).

Now pretrain the same ViTs on JFT-300M, a Google dataset of about 300 million labeled images, and the order flips. With the ViT paper's final fine-tuning recipe (its Table 2), ViT-L/16 reaches 87.76%, and ViT-H/14, the Huge size with 14-pixel patches, reaches 88.55%. Both beat BiT-L at 87.54%, a large ResNet pretrained on the same data (Kolesnikov et al. 2020). Removing knowledge hurt with a million images and helped with three hundred million.

This section explains that reversal exactly, and interviewers ask about it in many forms. What does a CNN know that a ViT does not? When is that knowledge worth having? Is a ViT really more robust? We go in five steps. We define an inductive bias and show, in a toy with exact numbers, why a true one is worth data. We state exactly which shifts of an image a convolution and a ViT respect, and prove it. We measure what the missing assumption costs at real scale. We look at what trained ViTs learn in its place, and prove that attention can compute any convolution. And we end with ways to put the assumption back and with the robustness evidence, read carefully.

An inductive bias is an assumption that picks the answer

A learner sees finitely many examples and must still answer for inputs it has never seen. Many different functions fit the same training set, so something besides the data has to pick one. Take three points on a line, $(0, 0)$, $(1, 1)$, and $(2, 2)$. A learner that fits only straight lines returns $y = x$. A learner that fits any polynomial may return $y = x + 5x(x - 1)(x - 2)$, which passes through the same three points but predicts 33 at $x = 3$ instead of 3. The data did not make that choice. The assumption "the answer is a line" did.

In these terms, a CNN's locality and weight sharing are hard biases. A ViT has far fewer hard biases about space, and its training recipe adds soft ones back (§27.4).

Why a true assumption is worth data

A larger class contains every function a smaller one does. So how can shrinking it help? The answer is that the learner never gets the best function in its class. It gets an estimate fit to finite, noisy data. The error of that estimate has two parts, and shrinking the class moves them in opposite directions.

In plain words, the approximation error is what the class gets wrong even with infinite data. The estimation error is what we lose by learning from the data we have. Shrinking the class to $\mathcal{H}' \subset \mathcal{H}$ can only raise the first term, because the best member of a smaller set is no better. It usually lowers the second, because fewer free parameters are fit to the same noise. This is the bias-variance trade-off of §0.7, stated for whole classes of models instead of one predictor.

Two cases follow. Call a restriction true when the best function really lies in $\mathcal{H}'$. A true restriction leaves the first term where it was and lowers the second, so it is pure gain. A restriction that is only approximately true wins while estimation error dominates, which is when data is scarce. It loses once data is so plentiful that the approximation error is all that remains. That is the argument of this whole section in one identity. The rest measures the two terms for images.

A toy where you can compute what an assumption is worth

Here is the trade-off with numbers you can check. Each input is a row of 32 random numbers. The target at each position is a local slope, the left neighbor minus the right neighbor, times a gain that drifts slowly along the row, plus noise. One model assumes the rule is identical at every position. The other assumes nothing. The first is slightly wrong, and the second has much more to learn.

Precisely, the input is $\mathbf{x} \in \R^n$ with $n = 32$ entries drawn independently from $\mathcal{N}(0, 1)$. Indexing is circular, so position $-1$ means position $n - 1$ and the row wraps around. The target at position $t$ is

with gain strength $\beta = 0.3$ and noise level $\sigma = 0.5$. The difference $x_{t-1} - x_{t+1}$ is a convolution with the kernel $(1, 0, -1)$, the slope detector of §0.8. The gain $1 + \beta g_t$ is 1.3 at position 0, 1.0 at position 8, and 0.7 at position 16, so the true rule is close to the same everywhere but not exactly. We fit two linear models by least squares, the weights that minimize the squared error on the training pairs, from $N$ training rows.

• The convolution has $k = 3$ weights shared by every position, $\hat y_t = w_0x_{t-1} + w_1x_t + w_2x_{t+1}$. It can represent only maps that apply one three-tap kernel everywhere.
• The full map is an arbitrary linear map $\hat{\mathbf{y}} = \mathbf{x}\mathbf{W}$ with $\mathbf{W} \in \R^{n\times n}$, 1,024 weights. Its class contains the true map.

We can compute both error terms for both models.

Approximation error. The full map has none, since the true map is in its class. For the convolution, the best kernel is the true one, $(1, 0, -1)$, because the drifting part averages out ($\sum_t g_t = 0$). What it misses at position $t$ is $\beta g_t(x_{t-1} - x_{t+1})$, whose variance is $2\beta^2 g_t^2$, since $x_{t-1}$ and $x_{t+1}$ are independent with variance 1. Averaged over positions, with $\frac1n\sum_t g_t^2 = \frac12$, that is $\beta^2 = 0.09$.

Estimation error. This term needs one standard fact about least squares.

In plain words, every coefficient you fit costs about $\sigma^2/N$ of test error. More coefficients cost more, and more equations cost less. The two models differ in both counts.

• The full map solves a separate regression for each of the 32 outputs, each with $p = n = 32$ coefficients and $N$ equations. Its excess per output is $\sigma^2 n/(N - n - 1)$.
• The convolution's three weights are shared by all 32 positions, so each training row gives it $n$ equations instead of one. Its excess is about $\sigma_{\text{eff}}^2\,k/(Nn)$, where $\sigma_{\text{eff}}^2 = \sigma^2 + \beta^2$, because the part it cannot represent acts like extra noise.

Add the noise floor $\sigma^2$, and the two models' test errors per output are about

In plain words, the convolution reaches its floor almost at once and stays there, a little above the noise. The full map starts much worse and keeps improving toward the noise itself. They cross where $\sigma^2 n/(N - n - 1) = \beta^2$, that is at $N^\star = n + 1 + \sigma^2 n/\beta^2 = 33 + 0.25 \cdot 32/0.09 \approx 122$ training rows.

The same formulas say how much data shared weights are worth when their assumption is exactly right.

A ViT sits between these two extremes. Its patch embedding is shared across all patch positions, and its attention and MLP weights are shared across all tokens, so it is nothing like a dense layer on pixels. What it lacks is one specific assumption, that a token's update depends mostly on its spatial neighbors. The rest of the section measures what that one assumption is worth. In an interview, the split into approximation and estimation error answers "why would a model with fewer assumptions ever win?" It wins once data shrinks its estimation error below the approximation error that its rival's assumptions cost.

Which shifts a convolution and a ViT respect

A model can respond to a moved image in two ways. It can move its output the same way, which is called being equivariant. A map of where the edges are should move when the edges move. Or it can leave its output unchanged, which is called being invariant. The label "cat" should not change when the cat moves. §0.8 proved that a stride-1 convolution is equivariant, and that averaging a feature map over all positions turns that into an invariant summary. Here we need the exact version, because it says precisely what a ViT gives up.

Start with one short signal and three layers you can check by hand. A stride-2 layer computes the same outputs as a stride-1 layer and keeps every second one. A patch embedding cuts the signal into pieces that do not overlap and maps each piece to one number.

That example contains the whole story. A stride-1 convolution respects every shift. A stride-$s$ layer respects only shifts by multiples of $s$. A patch embedding respects only shifts by whole patches. Now the general statements. We write an image as $\mathbf{X} \in \R^{H\times W\times C}$, with height $H$, width $W$, and $C$ channels, channels last in the math as in §27.1. Code uses PyTorch's $(C, H, W)$ and says so. Rows are indexed by $r$ and columns by $c$.

Circular shifts are a mathematical convenience. They form a group, since shifting by $\boldsymbol\delta$ and then by $\boldsymbol\delta'$ is shifting by $\boldsymbol\delta + \boldsymbol\delta'$, so every statement below is exact. Real images have borders, and zero padding breaks equivariance in a band of width $\lfloor k/2\rfloor$ along them. Everything we prove holds everywhere with circular padding, and away from the borders with zero padding.

We use the convolution of §0.8 in its cross-correlation form, the one deep learning libraries compute, with circular padding and a centered $k\times k$ kernel ($k$ odd, half-width $a = \lfloor k/2 \rfloor$). The weights are $\mathbf{W}_{\text{conv}} \in \R^{C_{\text{out}}\times C\times k\times k}$ in PyTorch's layout, and the bias is $\mathbf{b} \in \R^{C_{\text{out}}}$. Write $c'$ for the input channel, to keep it apart from the column $c$, and $o$ for the output channel, both counted from 0. Then

In plain words, output channel $o$ at pixel $(r, c)$ is a weighted sum of the $k\times k$ neighborhood around that pixel, with the same weights at every pixel. The output channel comes last here, so that one shift operator $\mathcal{T}_{\boldsymbol\delta}$ acts on inputs and outputs alike. (§27.1 put it first, as PyTorch does. The entries are the same numbers.) A convolution with stride $s$, for $s$ dividing $H$ and $W$, is the stride-1 output kept on every $s$-th row and column, $\operatorname{Conv}_s(\mathbf{X})[r, c, :] = \operatorname{Conv}(\mathbf{X})[sr, sc, :]$ for $0 \le r < H/s$ and $0 \le c < W/s$.

In plain words, a stride-$s$ layer is equivariant only to shifts that move its sampling grid onto itself. Stack such layers and the periods multiply, just as strides multiply in the receptive-field arithmetic of §0.8. ResNet-50's final feature map has a total stride of 32. So even with circular padding, the network is guaranteed to be equivariant only to shifts by multiples of 32 pixels, and its globally pooled output is guaranteed to be invariant only to those.

This is not a technicality. Shifting an image by a single pixel often changes a trained CNN's prediction (Azulay and Weiss 2019). The cause is aliasing, which a four-number example shows. Take the row $(1, -1, 1, -1)$ and keep every second entry. You get $(1, 1)$. Shift the row by one pixel first and you get $(-1, -1)$. Detail finer than the coarse grid can hold has turned into a false constant whose sign depends on where the grid falls. The sampling theorem says to blur before subsampling, and standard CNNs do not.

Zhang (2019) traced the lost equivariance layer by layer. It holds with period 1 before the first downsampling, and the period doubles at each one. Adding a low-pass filter before every subsampling restored much of it. ResNet-50's predictions under shifts became more consistent by up to 2.1 points, and its ImageNet accuracy rose by 0.7 to 0.9 points. In practice this is why random-crop augmentation matters even for CNNs. In an interview, "a CNN is translation invariant" is a wrong answer.

The symmetry a ViT keeps

A patch embedding is a convolution whose kernel size and stride both equal the patch size $P$ (§27.1). So part (b) of the theorem says it respects only shifts by whole patches. Such a shift moves every patch to another slot of the patch grid. The ViT receives the same tokens in a different order, and a Transformer without position information does not care about order.

Here is that in the smallest case. Take a 2 by 2 grid of patches, numbered 0 and 1 in the top row and 2 and 3 below, with tokens $(\mathbf{t}_0, \mathbf{t}_1, \mathbf{t}_2, \mathbf{t}_3)$. Shift the image right by one whole patch, circularly. Patch 0's content moves to slot 1, patch 1's wraps around to slot 0, and the same happens in the bottom row. The sequence becomes $(\mathbf{t}_1, \mathbf{t}_0, \mathbf{t}_3, \mathbf{t}_2)$, the same four tokens reordered.

In general, the patch grid has $H_p = H/P$ rows, $W_p = W/P$ columns, and $N_p = H_pW_p$ patches. Stack the flattened patches in raster order (patch $(r, c)$ in row $rW_p + c$) as $\mathbf{X}_p \in \R^{N_p\times P^2C}$. The patch tokens are then $\operatorname{PE}(\mathbf{X}) = \mathbf{X}_p\mathbf{W}_E + \mathbf{1}\mathbf{b}_E^\top \in \R^{N_p\times d}$, with weights $\mathbf{W}_E \in \R^{P^2C\times d}$ and bias $\mathbf{b}_E \in \R^d$, as in §27.1. A translation by $\boldsymbol\delta \in P\mathbb{Z}^2$ moves whole patches, so it acts on the token matrix as a permutation,

where $\boldsymbol\Pi_{\boldsymbol\delta}$ is the $N_p\times N_p$ permutation matrix that sends grid cell $(r, c)$ to $\big((r + \delta_r/P)\bmod H_p, (c + \delta_c/P)\bmod W_p\big)$.

Two facts from §4.6 finish the argument. First, a Transformer encoder $f$ without positions is permutation equivariant, $f(\boldsymbol\Pi\mathbf{X}) = \boldsymbol\Pi f(\mathbf{X})$ for every permutation matrix $\boldsymbol\Pi$. Reordering the tokens reorders the outputs and changes nothing else. Second, a fixed matrix $\mathbf{B}$ added to every head's attention scores keeps that property for every permutation with $\boldsymbol\Pi\mathbf{B}\boldsymbol\Pi^\top = \mathbf{B}$. The scores still transform as $\boldsymbol\Pi\mathbf{S}\boldsymbol\Pi^\top$, which is all the proof in §4.6 uses, and it is the same reason a mask preserves the symmetries that leave it unchanged.

In plain words, a ViT's symmetry lives on the patch grid, never on the pixel grid. With relative positions, or none, it is exactly equivariant to shifts by whole patches. With learned absolute positions, the ViT paper's default, it is not even that. The content moves to new slots while the position vectors stay with the slots, so the moved content meets different positions. Any approximate equivariance the model shows was learned from data. Exact equivariance would force all patch rows of $\mathbf{P}_{\text{pos}}$ to be equal, which is the same as having no positions.

This is the precise content of the remark in the ViT paper that only its MLPs are local and translation equivariant. The MLPs act on each token separately, so they commute with every reordering. The attention layers are global, and their only spatial structure comes from positions the model must learn.

The cell measures all of this on a random image, for horizontal shifts of 0 to 8 pixels with $P = 8$. When the stride $s$ divides the shift, the error is $\max\lvert f(\mathcal{T}_{\boldsymbol\delta}\mathbf{X}) - \mathcal{T}_{\boldsymbol\delta/s}f(\mathbf{X})\rvert / \max\lvert f(\mathbf{X})\rvert$. Otherwise it is the smallest such error over every possible output shift, so a nonzero value means no output shift works.

The stride-1 convolution has zero error at every shift. The stride-2 convolution is exact at even shifts, and at odd ones it is wrong by more than the size of its own output (an error of 1.12). The patch embedding is exact only at 0 and 8 pixels. Adding positions breaks even the shift by a whole patch, with an error of 0.85 at 8 pixels. The widget shows the same pattern on a picture. Pick a model and a patch size, then move the shift slider.

import numpy as np

def equivariance_error(f, x, shift, stride=1):
    raise NotImplementedError

def which_shifts_commute(f, x, stride=1, tol=1e-8):
    raise NotImplementedError

The data-scale crossover, measured

The toy predicted a crossover, and the symmetry theorem says exactly what a ViT gives up. The ViT paper measured the crossover directly, in two experiments.

In the first, the same ViTs were pretrained on three datasets of growing size and then fine-tuned on ImageNet at 384 by 384. The middle dataset is ImageNet-21k, about 14 million images in about 21,800 classes. Their Table 5 gives the top-1 accuracies.

Read the columns from top to bottom. On ImageNet alone the bigger model is worse, which is the signature of estimation error. With 14 million images the sizes are close. Only with 300 million does size pay off. The convolutional BiT models that win on the small datasets are overtaken there (their Figure 3).

The second experiment separates the model from the training recipe. They pretrained on random subsets of JFT of 9M, 30M, 90M, and 300M images, with the same hyperparameters for every size. The small subsets got no extra regularization, only early stopping. Then they measured 5-shot linear accuracy on ImageNet. The network is frozen, a linear classifier is fit on its features from five labeled images per class, and that classifier is tested. Nothing is fine-tuned, so the score measures the pretrained features alone.

The same paper compared three families at equal compute, with data no longer the bottleneck because all were pretrained on JFT-300M. The third family is a hybrid, the first stages of a ResNet feeding their feature map, as tokens, to a ViT. Their Table 6 lists ImageNet accuracy against total pretraining compute in exaFLOPs ($10^{18}$ operations). The cell plots it. For each ResNet it then asks how much compute the ViTs need to reach the same accuracy. It interpolates linearly in log compute between the two nearest ViTs on the frontier, the ViTs that beat every cheaper ViT.

Two later results keep this from becoming a slogan. Purely convolutional networks pretrained on four billion images match ViTs at comparable compute, and a hybrid of convolution and attention matches a JFT-pretrained ViT-H/14 with 23 times less data. The box gives the numbers.

So the honest summary has two parts. A good prior is worth a large amount of data. At very large scale, the choice between a well-tuned CNN and a ViT matters much less than compute. ViTs won the frontier mostly for other reasons. They scale smoothly on accelerators. The same tokens feed language models and multimodal systems (CLIP's image tower is a ViT). And masked and self-supervised pretraining fit them naturally, which is another way to buy what labels cannot (self-supervised vision). An interviewer who asks "when does a ViT beat a ResNet?" wants both parts, the crossover with its numbers and the caveat that well-tuned CNNs at scale close the gap.

What trained ViTs learn in place of locality

A CNN's early layers look at small neighborhoods because they have no choice. A ViT's attention can reach anywhere. If a ViT must learn locality from data, does it? The ViT paper answered with one number per attention head, how far away, in pixels, the head's attention lands on average.

To define it we need the grid position of each token. The class token, and any register tokens (extra learned tokens that give the model scratch space, §27.4), come first and have no position. We call them prefix tokens. The patches follow in raster order.

In plain words, for every patch, average how far away its attention lands, weighted by the attention weights. Then average over patches and images. It is the attention analog of a receptive field size, with one difference. A receptive field is where a unit can look. Attention distance summarizes where the weights do point.

Measured on trained models, the statistic tells a consistent story.

Two more findings sharpen the picture. In the hybrid R50+ViT-L/16, where a ResNet has already done the local processing, the early local heads are much less pronounced. That suggests those heads do the job early convolutions do. And ViT-B/32 still learns local heads when trained on ImageNet alone. It also depends less on large-scale pretraining. In the ViT paper's Table 5, JFT-300M pretraining adds 7.4 points to ViT-B/32 on ImageNet and 10.6 points to ViT-L/16. Raghu et al. also found that lower ViT layers are most similar to lower ResNet layers when only the most local heads are kept. Put together, locality is a useful thing to compute early. A ViT learns it when it has enough data, and failing to learn it goes with failing to train well.

import numpy as np

def mean_attention_distance(A, H_p, W_p, P, n_prefix):
    raise NotImplementedError

One caution before you read too much into such plots. Attention distance summarizes weights, not influence. A head can put large weight on a token whose value vector is small. And the residual stream of §5.1 carries every token's own information around each attention layer. Influence measures can tell a different story. The most direct ones are gradients of an output with respect to the input patches, like the effective receptive field of §0.8. Attention rollout, next, is a cheap compromise that stays within the attention maps. Use MAD for what it is, a cheap and comparable summary of where single heads look.

Attention rollout, following weights through the layers

A single layer's attention map says where a token looked in the layer below. But the token it looked at had already mixed in other tokens in earlier layers. And every token also passes its own state up the residual path. Abnar and Zuidema (2020) proposed following both effects through the whole network.

In plain words, treat each layer as moving information along the edges of a graph, half through attention and half straight up the residual path. Multiply the weights along every path from an input patch to the class token at the top, and add over paths. The order of the product follows from writing tokens as rows. If layer $\ell$ mixes its input rows as $\mathbf{X}^{(\ell)} \approx \hat{\mathbf{A}}^{(\ell)}\mathbf{X}^{(\ell-1)}$, then $\mathbf{X}^{(L)} \approx \hat{\mathbf{A}}^{(L)}\cdots\hat{\mathbf{A}}^{(1)}\mathbf{X}^{(0)}$, with the last layer on the left.

The $\frac12$ comes from the residual connection. A layer computes $\mathbf{X} + \bar{\mathbf{A}}\mathbf{X} = (\bar{\mathbf{A}} + \mathbf{I})\mathbf{X}$, and the paper divides by 2 so that the rows sum to 1, which gives the residual path and attention equal shares. Attention rows are nonnegative and sum to 1, so $\frac12\bar{\mathbf{A}} + \frac12\mathbf{I}$ already has rows that sum to 1, and so does the product. The renormalization matters only for maps whose rows do not, for example after a visualization tool zeroes the smallest weights. The class token's patch entries then sum to 1 minus the share that ends on prefix tokens.

Rollout is a heuristic, not a measurement of influence. It ignores the value vectors and the MLPs, averages the heads, and fixes the residual's share at one half. So, like a raw map, it cannot see that a heavily weighted token may carry little. Unlike a single map, it accounts for depth and for the residual path. Abnar and Zuidema tested it on a small six-layer text classifier. From the second layer up, rollout agreed better than raw attention with two other measures of token importance, blanking out input tokens and input gradients. In the first layer, raw attention agreed better. It is popular because it costs $L$ matrix products and needs no backward pass. Methods that also propagate gradients were built to address what it leaves out (Chefer, Gur and Wolf 2021).

import numpy as np

def attention_rollout(A_layers, n_prefix, H_p, W_p):
    raise NotImplementedError

Attention can compute any convolution

Early ViT heads that attend to a few nearby patches look a lot like the taps of a convolution. How far does the resemblance go? A ViT lacks a convolution's built-in symmetry, but is it unable to compute what a convolution computes? No. Attention can compute any convolution exactly, in a limit, and the proof is a construction (Cordonnier, Loukas and Jaggi 2020).

The idea fits in three sentences. A $3\times3$ convolution adds up nine terms, each one neighbor of the pixel times one weight matrix, at a fixed offset. An attention head whose scores depend only on the offset from query to key can put almost all its weight on one neighbor at a fixed offset. Nine such heads, one per offset, each with its value and output weights set to that offset's kernel tap, add up to the convolution.

Here is one head in numbers, in 1-D. It should read the neighbor one step to the right, offset $+1$. Score each key by $-\alpha(\delta - 1)^2$, where $\delta$ is the key's offset from the query and $\alpha$ is a sharpness. With $\alpha = 2$, the target key gets score 0 and its two neighbors get $-2$, so the softmax puts $1/(1 + 2e^{-2}) \approx 0.79$ on the target and 0.11 on each neighbor. With $\alpha = 10$, it puts 0.99991 on the target. As $\alpha$ grows, the head becomes a lookup of exactly one neighbor.

Now the general case. First, write multi-head attention so the heads are visible. Take $h$ heads with per-head value projections $\mathbf{W}_V^{(j)} \in \R^{C\times d_v}$, and the output projection $\mathbf{W}_O \in \R^{hd_v\times d_{\text{out}}}$ split into row blocks $\mathbf{W}_O^{(j)} \in \R^{d_v\times d_{\text{out}}}$. Weights are stored (input, output) and applied to row-vector tokens, as in §4.5. The output for query token $q$ is

In plain words, concatenating the heads and multiplying by $\mathbf{W}_O$ is the same as multiplying each head by its own block and adding. So each head contributes its attention-weighted inputs times one matrix $\mathbf{W}^{(j)}$, of rank at most $d_v$. Now write the convolution the same way. Let $\mathbf{W}_{\boldsymbol\Delta} \in \R^{C\times C_{\text{out}}}$ be the kernel tap at offset $\boldsymbol\Delta = (u - a, v - a)$, with entries $\mathbf{W}_{\boldsymbol\Delta}[c', o] = \mathbf{W}_{\text{conv}}[o, c', u, v]$. Then

The two formulas have the same shape. A convolution is $k^2$ "heads" that each read one fixed neighbor. So the only question is whether an attention head can read exactly one neighbor at a fixed offset, for every query.

Two limits keep the theorem honest. First, the sizes are real. ViT-B has 12 heads of width 64, so the construction fits at most a 3 by 3 convolution (nine heads) with 64 output channels into one of its layers, not a full 768-channel one. Second, expressing is not learning. The theorem says the class of attention layers contains the class of convolutions, which is exactly the sense in which a ViT has the weaker inductive bias. Whether gradient descent finds the convolution is an empirical question. Cordonnier et al. trained a network of six attention-only layers with nine heads each on CIFAR-10 images, after a 2 by 2 space-to-channel downsampling. Some heads in the first two layers did learn to attend to single positions at fixed offsets from the query, both with the quadratic encoding and with a freely learned one.

How fast does the construction converge? The one-dimensional version needs one head per tap and a two-number encoding $(\delta^2, \delta)$. The cell builds it for a three-tap kernel on a circular signal and compares it with the convolution as $\alpha$ grows. Next to the error it prints the bound's $\rho(\alpha)$, which in 1-D is $2\sum_{z\ge1}e^{-\alpha z^2}$.

At $\alpha = 1$ the heads are still blurry, and the error is as large as the outputs themselves. By $\alpha = 10$ it has fallen to the $10^{-4}$ range, and by $\alpha = 20$ it is near $10^{-8}$. It falls like $\rho(\alpha) \approx 2e^{-\alpha}$, as the bound predicts. In float32, whose rounding unit is $2^{-24} \approx 6\times10^{-8}$, a sharpness of about 18 already makes the off-target weights invisible, so a moderate, finite $\alpha$ is enough in practice. The widget shows the two-dimensional version, nine heads around one pixel. Press Sharpen, or raise the sharpness by hand, and watch each head's weight collapse onto one neighbor.

The practical reading is short. A ViT layer is strictly more expressive than a convolution of the same width, so whatever a CNN layer computes, a ViT layer with relative positions can compute too. What it lacks is the guarantee that training will find that solution from little data. That distinction, expressive power against inductive bias, is the heart of the interview question "can self-attention represent a convolution?"

import numpy as np

def conv_via_attention(X, W_conv, alpha):
    raise NotImplementedError

Putting the assumption back, and what "no inductive bias" gets wrong

A ViT is often described as having "no inductive bias". That is false, and the correction is a useful interview answer. Cutting the image into patches is a strong locality assumption at the scale of $P$ pixels. The patch embedding is shared across positions, so it is equivariant to whole-patch shifts. The MLPs treat every token identically. When the resolution changes, the position table is resized by 2-D interpolation, an explicit statement that positions live on a grid (§27.1). And the residual architecture, the optimizer, and the initialization all prefer some functions over others. What a ViT lacks is the specific prior that a token's update depends mostly on its spatial neighbors, and exact translation symmetry at pixel resolution.

When data is limited, there are four ways to buy the missing prior back, and the rest of this chapter uses all of them.

1. Hard locality. Mask attention so each patch sees only patches within a small grid distance, or attend inside local windows. This brings back the convolution's restricted class, and it cuts the cost from quadratic to linear in the number of tokens. Swin's shifted windows and ViTDet's windowed blocks, in §27.5, are this idea. The 1-D version for language is the sliding window of §12.6.
2. Soft locality. Initialize or bias attention toward local offsets and let training undo it if the data disagree. Relative position biases also restore the whole-patch symmetry of the theorem above, exactly for circular offsets and away from the border otherwise.
3. Convolutional stems and hybrids. Let convolutions process the first few stages and attention the rest, as in the ViT paper's hybrids and in CoAtNet.
4. Substitute data for the prior. Use heavy augmentation, regularization, and distillation from a convolutional teacher, which is how DeiT trained a ViT on ImageNet-1k alone (§27.4). Or pretrain on unlabeled images with a self-supervised objective.

The cleanest controlled measurement of a soft prior at small scale comes from ConViT (d'Ascoli et al. 2021). Its attention layers start as convolutions, with each head initialized to attend around one offset, as in the theorem. A learned gate per head then decides how much of this positional attention to keep and how much content-based attention to mix in. The experiment compares ConViT-S with DeiT-S, a ViT of the Small size (22 million parameters) trained with the DeiT recipe. Both are trained with that recipe on class-balanced subsets of ImageNet-1k, with epochs scaled so the number of images seen stays constant.

The table puts the options side by side.

The first route is a mask. Building it correctly, with the prefix tokens left free, is a small but common interview task.

import numpy as np

def grid_local_mask(H_p, W_p, radius, n_prefix=1):
    raise NotImplementedError

Robustness and shape bias, read carefully

The claim you will hear most often is that ViTs are "more robust" than CNNs. Some of the evidence is striking, and some of it dissolves under a controlled comparison. You need both halves for an interview.

The best-known behavioral test asks whether a model recognizes objects by their shape or by their texture. Geirhos et al. (2019) used style transfer to make 1,280 cue-conflict images in 16 categories. Each has the shape of one category and the texture of another, such as a cat's outline in elephant skin. The test is which category a model, or a person, picks.

The definition is a two-line count. Here it is on six hand-made trials. In the fourth the model names neither the shape nor the texture, so it does not count, and three shape decisions against two texture decisions give a shape bias of 0.60.

Before you credit that 87% to attention, look at two facts from the same literature. Geirhos et al. trained the same ResNet-50 on Stylized-ImageNet, where each image's texture is replaced by a random painting style, and its shape bias rose from 22% to 81%. And ViT-22B differs from a typical model in data (billions of images), size, and training, not only in architecture. Shape bias responds strongly to data and objective, so a high number tells you less about the architecture than it seems to.

Occlusion gives the same lesson. Naseer et al. (2021) zeroed random 16 by 16 patches of ImageNet validation images. With half the image removed, DeiT-S kept 70% top-1 accuracy while ResNet-50 fell to 0.1%, and DeiT-B still had 37% with 90% removed. That is a real property of the models as trained. But three things favor the ViT. The occlusion is aligned to the ViT's own patch grid. A zeroed patch is one constant token to a ViT but a set of sharp artificial edges to a convolution. And DeiT was trained with CutMix and random erasing, augmentations that paste over or blank out image regions, which a standard ResNet-50 recipe does not use.

Bai et al. (2021) ran the controlled experiment. They compared DeiT-S (22 million parameters, 76.8% on ImageNet) with ResNet-50 (25 million, 76.9%) and swapped training ingredients between them. Three kinds of robustness came out differently. Adversarial robustness is accuracy under small worst-case pixel changes. Patch attacks paste small adversarial texture patches onto the image. And out-of-distribution sets test photos unlike the training data. Two standard ones are ImageNet-A, natural photos that standard classifiers get wrong, and ImageNet-C, photos with synthetic corruptions (noise, blur, weather, digital artifacts) scored by mean corruption error (mCE, lower is better). The box gives the result and the numbers behind it.

So a matched comparison turns two of the three famous robustness gaps into differences of recipe. The lesson reaches well beyond this one paper, because most robustness claims in the literature change the architecture and the recipe at the same time.

In PyTorch

Here are the same symmetry facts, checked with real PyTorch modules. A Conv2d with padding_mode="circular" commutes with torch.roll. A ViT-style stack, a Conv2d patch embedding with kernel and stride 8 followed by a two-layer pre-LN TransformerEncoder, commutes with a shift by one whole patch when it has no position embeddings. The output tokens move one grid column and are otherwise unchanged. Add an absolute position table and that equivariance is gone. Shifts by a fraction of a patch match no grid shift at all.

Read more

• Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Section 3.1 states the inductive-bias argument, Section 4.3 and Figure 4 measure the data crossover, Table 6 and Figure 5 compare compute, and Figure 7 and Appendix D.7 introduce mean attention distance.
• Cordonnier, Loukas & Jaggi (2020). On the Relationship between Self-Attention and Convolutional Layers. Theorem 1 and Lemmas 1 and 2 prove that attention can express convolution, and the experiments show early heads learning grid-like patterns.
• Raghu et al. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. Representation similarity between ViTs and ResNets, attention distances with and without large-scale data, and the role of skip connections.
• d'Ascoli et al. (2021). ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. A gated, convolution-initialized attention layer, and the cleanest data-subsampling measurement of what a soft prior is worth.
• Abnar & Zuidema (2020). Quantifying Attention Flow in Transformers. Attention rollout and attention flow, the residual correction $\frac12\mathbf{A} + \frac12\mathbf{I}$, and a comparison with ablation and gradient importance.
• Zhang (2019). Making Convolutional Networks Shift-Invariant Again. How strided layers lose equivariance, layer by layer, and how anti-aliasing restores it.
• Bai et al. (2021). Are Transformers More Robust Than CNNs?. The matched comparison of DeiT-S and ResNet-50 that separates architecture from recipe.
• Geirhos et al. (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. The cue-conflict test, the shape-bias measure, and Stylized-ImageNet.

Watch next

▶