Chapter 27 · Section 27.5 · 80 min

Hierarchical and efficient vision backbones

A vision backbone is the network that turns an image into feature vectors for a task head to read. A classifier needs one vector for the whole image, but a detector or a segmenter needs a vector for every region, at several sizes, because the objects it must find come in every size. This section builds backbones that give exactly that, a stack of feature maps from fine to coarse, at a cost that grows only linearly with the number of pixels. Almost every detection, segmentation, and video model you will build or be asked about sits on one of them.

Start with what a plain Vision Transformer (ViT) gives you. §27.2 built it. It cuts the image into 16 by 16 pixel patches, turns each patch into a token, and runs a stack of identical Transformer blocks on the sequence. It also counted the cost in multiply-adds (MACs). One multiply-add is one multiplication plus one addition, and it is the unit that vision papers report as "FLOPs". A block on $T$ tokens of width $d$ costs $12Td^2$ multiply-adds in its linear layers, the four attention projections and an MLP that widens to $4d$ . Attention itself, the scores and the weighted sum, costs $2T^2d$ .

Now ask ViT-B/16, the base model with width 768 and 12 blocks, to segment a 1024 by 1024 street scene. The cell adds up the cost at three resolutions.

Python · cell 1

So the question is whether a Transformer can give what the ResNet gives, maps at several scales for a cost linear in the image area. This section answers yes, in three ways, and then asks which to choose.

The Swin Transformer attends only inside small windows, shifts the windows between layers so that information crosses their borders, and merges neighboring tokens between stages.
ConvNeXt asks how much of Swin's success came from attention at all. It rebuilds a ResNet with Transformer-era choices until it matches Swin.
Hybrids put convolutions early in the network and attention late, each where it works best.

Then come two later results. With strong pretraining on unlabeled images, a plain ViT with a few windows becomes the best dense backbone again. And a trained plain ViT can run about twice as fast if you merge similar tokens instead of changing its attention. The section ends with why counted FLOPs do not predict speed, and how to choose a backbone.

Why dense prediction wants a pyramid

Dense prediction means producing an output at every location of the image. Detection draws a box around each object and names it. Segmentation gives every pixel a label. Both read a feature map, a grid of vectors in which each vector describes one region of the image. The stride of a map is the number of pixels between neighboring vectors, so a map at stride 32 has one vector per 32 by 32 block of pixels.

Objects come in every size. On COCO, the standard detection benchmark of everyday photos with boxes and masks for 80 kinds of object, a car can be 20 pixels wide or 600. A map at stride 32 carries rich context but cannot place a 20-pixel object, which is smaller than one of its cells. A map at stride 4 is precise, but each of its vectors has seen only a small neighborhood, unless the network runs many expensive layers at that resolution. The answer is a pyramid of maps, fine ones for small objects and coarse ones for large ones.

Convolutional networks build that pyramid as they go. Recall from §0.8 that a ResNet is a stack of convolutional blocks with residual connections. Its first layers, its stem, turn a 224 by 224 image into a 56 by 56 map. Its four stages then produce maps of 56, 28, 14, and 7 cells a side, at strides 4, 8, 16, and 32, with more channels as the maps shrink. A detection head such as a feature pyramid network (Lin et al. 2017) reads all four maps and combines them, and §27.6 builds one.

📐 Hierarchical backbone

A hierarchical (multi-scale) backbone maps an image $\mathbf{X}_{\text{img}} \in \R^{H\times W\times C}$ , of height $H$ , width $W$ , and $C$ channels, to a list of feature maps, one per stage $s = 1, \dots, S$ ,

\mathbf{F}_s \in \R^{H_s\times W_s\times d_s}, \qquad H_s = \frac{H}{4\cdot 2^{s-1}}, \quad W_s = \frac{W}{4\cdot 2^{s-1}}, \quad d_s = 2^{s-1}d_1 .

With $S = 4$ stages the strides are 4, 8, 16, and 32, and the widths are $d_1, 2d_1, 4d_1, 8d_1$ . A plain (isotropic) backbone such as ViT keeps one map, $H/P \times W/P \times d$ for patch side $P$ , through every layer.

In plain words, each stage halves the map in both directions and doubles the channels. A small count shows why those two go together. A block whose cost is dominated by per-token linear layers costs about $c\,H_sW_sd_s^2$ for some constant $c$ . At stage 1 of a 224-pixel image with $d_1 = 96$ , $H_sW_sd_s^2 = 56\cdot56\cdot96^2 \approx 28.9$ million. At stage 2 it is $28\cdot28\cdot192^2$ , again 28.9 million, because the map lost a factor of 4 and $d_s^2$ gained a factor of 4. So one block costs the same in every stage, and designers spend compute across scales by choosing how many blocks each stage gets.

A plain ViT misses both halves of this design. It keeps one map, so a detector must invent the other scales. And its cost per block, $12Td^2 + 2T^2d$ , has $T$ set by the finest resolution it must see, so the quadratic term eventually dominates. The Swin Transformer (Liu et al. 2021) fixes the cost with windows and the single map with patch merging, and we take them in that order. An interviewer often opens with "why can't a detector read a plain ViT's output as it is?" The two answers are the missing scales and the quadratic cost.

Window attention, and what it costs

Window attention cuts the token map into small squares, called windows, and runs ordinary attention inside each one. A token sees only the tokens of its own window. Here is the saving on a map small enough to count.

Why squares that do not overlap? §12.6 made attention cheap for language with a sliding window, in which each token sees only the few tokens just before it. The 2-D version gives every query its own square of keys centered on itself. That is cheap in FLOPs but awkward on a GPU, because neighboring queries share most but not all of their keys, so the keys cannot be packed into one dense matrix multiply. With windows that do not overlap, each window is a small, dense attention problem, and all of them run together as one batched matrix multiply.

📐 Window multi-head self-attention (W-MSA)

Let the token map be $\mathbf{X} \in \R^{H_p\times W_p\times d}$ , with $T = H_pW_p$ tokens of width $d$ , and let $M$ be the window side in tokens, with $M$ dividing both $H_p$ and $W_p$ . Window $(a, b)$ holds the $M^2$ tokens at rows $aM, \dots, aM + M - 1$ and columns $bM, \dots, bM + M - 1$ , for $a = 0, \dots, H_p/M - 1$ and $b = 0, \dots, W_p/M - 1$ , so there are $n_w = T/M^2$ windows. Stack the tokens of each window in raster order (row by row, left to right) into $\mathbf{X}_{ab} \in \R^{M^2\times d}$ . W-MSA applies multi-head attention with $h$ heads to each window separately,

\mathbf{O}_{ab} = \operatorname{MHA}(\mathbf{X}_{ab}), \qquad \text{head } j:\ \operatorname{softmax}\!\left(\frac{\mathbf{Q}_j\mathbf{K}_j^\top}{\sqrt{d_k}} + \mathbf{B}_j\right)\mathbf{V}_j ,

with $\mathbf{Q}_j = \mathbf{X}_{ab}\mathbf{W}_{Q,j}$ , $\mathbf{K}_j = \mathbf{X}_{ab}\mathbf{W}_{K,j}$ , $\mathbf{V}_j = \mathbf{X}_{ab}\mathbf{W}_{V,j}$ (projections stored as (input, output) and applied to row-vector tokens, as everywhere in this chapter), $d_k = d/h$ , and $\mathbf{B}_j \in \R^{M^2\times M^2}$ a relative position bias, defined later in this section. All windows share the same weights.

The first piece of code every Swin implementation needs cuts the map into windows and puts it back together.

Interview

Window partition and its inverse

Implement window_partition(x, M). The input x is a batch of token maps of shape (B, H, W, d), and M divides both H and W. Return an array of shape (B * n_w, M * M, d), with $n_w = (H/M)(W/M)$ windows per image. Order the windows by image first, then by window row, then by window column, and list the tokens inside each window in raster order.

Then implement window_reverse(windows, M, H, W), which undoes the partition and returns (B, H, W, d). Use reshapes and one transpose in each function, and no Python loops. The classic bug is a single reshape with no transpose. It runs without error but mixes rows of different windows.

import numpy as np

def window_partition(x, M):
    B, H, W, d = x.shape
    return x.reshape(-1, M * M, d)          # wrong: mixes rows of different windows

def window_reverse(windows, M, H, W):
    d = windows.shape[-1]

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Window attention is also global attention with a mask. Recall from §4.4 that an additive mask adds 0 to the scores that are allowed and $-\infty$ to the rest, so blocked keys get zero weight after the softmax. Here the mask is 0 between tokens of the same window and $-\infty$ otherwise. If you sort the tokens by window, the mask is block diagonal, one block per window. The cell runs both computations on the 8 by 12 map above and counts the scores each one computes.

Python · cell 2

The outputs agree exactly, and the windows compute 1,536 scores where global attention computes 9,216. In general the saving in the attention term is a factor $T/M^2$ , the number of windows. Here is the exact cost.

Here is where each term comes from. The four projections $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$ are each a $(T\times d)$ by $(d\times d)$ product, $Td^2$ multiply-adds each, so $4Td^2$ in total, the same in both layers. For the scores, head $j$ forms $\mathbf{Q}_j\mathbf{K}_j^\top$ . In global attention that is $T$ queries times $T$ keys times $d_k$ numbers, so $T^2d_k$ per head and $T^2d$ over the $h$ heads, since $hd_k = d$ . The weighted sum $\mathbf{A}_j\mathbf{V}_j$ , with $\mathbf{A}_j$ the attention weights, costs the same, which gives $2T^2d$ . In window attention each query meets only the $M^2$ keys of its window, so the two products cost $TM^2d_k$ each per head and $2M^2Td$ in total. $\blacksquare$

In plain words, windows leave the projections alone and replace the $T$ in the attention term by the window size $M^2$ . The cell evaluates both costs for the first stage of Swin-T, the smallest Swin model. Swin-T cuts the image into 4 by 4 pixel patches, embeds them at width $d = 96$ , and uses windows of $M = 7$ tokens.

Python · cell 3

The widget plots the cost of one whole block, attention plus MLP, against resolution, for global attention, window attention, a ConvNeXt block (met later in this section), and a dense convolution.

🤔 Predict before you slide

The widget is fixed at Swin-T's first stage, width 96 at stride 4. At 224 pixels a global block costs about 6 times a window block. The widget counts whole blocks, while the theorem counts only the attention layer. So add the MLP's $8Td^2$ to both sides, which gives $12Td^2 + 2T^2d$ against $12Td^2 + 2M^2Td$ . Predict the block ratio $(12d + 2T)/(12d + 2M^2)$ at 2048 pixels before you drag the slider there. Then raise the window size from 7 to 14. Which line moves, and by roughly how much at 2048?

Hint 1

At 2048 pixels and stride 4 the map is 512 by 512, so

T = 512^2 = 262{,}144

, and

d = 96

gives

12d = 1{,}152

. With

M = 7

the window term is

12d + 2M^2 = 1{,}250

Hint 2

The ratio is

525{,}440/1{,}250 \approx 420

. With

M = 14

the window term becomes

1{,}152 + 392 = 1{,}544

, so only the window line moves, up by a factor

1{,}544/1{,}250 \approx 1.24

, and the ratio falls to about 340.

In practice, window attention is what makes a Transformer affordable at stride 4. The cost of a block becomes a fixed multiple of the number of tokens, like a convolution's. The interview question is the derivation above, and the follow-up is "what did you lose?" The next subsection answers it.

Shifted windows let information cross borders

Windows have a flaw that the cost formula hides. A token never sees a token in the next window, in this layer or in any later one, because every layer uses the same windows. The receptive field of a token is the set of input tokens its output depends on. Stack as many window layers as you like, and every token's receptive field stays its own $M\times M$ window.

Swin's ablation, an experiment that removes one part and measures the damage, puts a price on that flaw (Liu et al. 2021, Table 4, Swin-T). It reports three scores. ImageNet top-1 is the share of images whose highest-scoring class is correct. COCO box AP measures detection on a scale of 0 to 100, and ADE20K mIoU measures how well predicted segments overlap the true ones on a scene-parsing benchmark. §27.6 defines both. Without the fix below, top-1 drops from 81.3% to 80.2%, box AP from 50.5 to 47.7, and mIoU from 46.1 to 43.3. The dense tasks suffer most.

The fix is to alternate two partitions. Odd layers use the regular windows. Even layers use windows moved by $\delta = \lfloor M/2\rfloor$ tokens down and to the right. With $M = 4$ , a window that covers rows 0 to 3 in one layer is followed by one that covers rows 2 to 5. Each new window straddles a border of the old ones. Here is what that does on one row of tokens.

✏️ Two layers on one row of eight tokens

Take a row of 8 tokens and windows of $M = 4$ , so $\delta = 2$ . Layer 1 uses the windows $\{0, 1, 2, 3\}$ and $\{4, 5, 6, 7\}$ . Layer 2 uses the shifted windows $\{0, 1\}$ , $\{2, 3, 4, 5\}$ , and $\{6, 7\}$ , where the two end pieces are cut short by the border.

Follow token 3. After layer 1 it depends on tokens 0 to 3. In layer 2 it sits in the shifted window $\{2, 3, 4, 5\}$ and attends to those four layer-1 outputs. Tokens 2 and 3 carry information from tokens 0 to 3, and tokens 4 and 5 carry information from tokens 4 to 7. So after two layers token 3 depends on all 8 tokens, $2M$ of them. Without the shift, layer 2 would use the window $\{0, 1, 2, 3\}$ again, and token 3 would still see only 4 tokens.

Swin uses pre-LN blocks, which apply LayerNorm (LN) at the start of each residual branch (§5.2). So a consecutive pair of Swin blocks is

\begin{aligned} \hat{\mathbf{X}}^{(\ell)} &= \mathbf{X}^{(\ell-1)} + \text{W-MSA}\big(\operatorname{LN}(\mathbf{X}^{(\ell-1)})\big), &\quad \mathbf{X}^{(\ell)} &= \hat{\mathbf{X}}^{(\ell)} + \operatorname{MLP}\big(\operatorname{LN}(\hat{\mathbf{X}}^{(\ell)})\big),\\ \hat{\mathbf{X}}^{(\ell+1)} &= \mathbf{X}^{(\ell)} + \text{SW-MSA}\big(\operatorname{LN}(\mathbf{X}^{(\ell)})\big), &\quad \mathbf{X}^{(\ell+1)} &= \hat{\mathbf{X}}^{(\ell+1)} + \operatorname{MLP}\big(\operatorname{LN}(\hat{\mathbf{X}}^{(\ell+1)})\big). \end{aligned}

Here $\mathbf{X}^{(\ell)}$ is the token map after block $\ell$ , and SW-MSA is window attention over the shifted partition.

How fast does the receptive field grow in general? Look at one axis. The regular windows are the intervals $[kM, kM + M)$ , and the shifted ones are $[kM + \delta, kM + \delta + M)$ . With $0 \lt \delta \lt M$ , every shifted window overlaps exactly two regular windows, and every regular window overlaps exactly two shifted ones. So each layer joins two neighboring windows of the layer before, as the example showed.

∂ Proof of the receptive-field theorem

Both partitions are products of a partition of the rows and a partition of the columns. So the set a token depends on is a product of a row interval and a column interval, and it suffices to prove the 1-D statement, that the dependency set has length $LM$ .

Every token of a window attends to every token of that window. The MLP, the norms, and the residual act on each token separately. So after layer $l$ , all tokens of a layer- $l$ window depend on the same set, the union of the sets of the layer- $(l-1)$ windows it overlaps.

For $l = 1$ that set is the window itself, an interval of length $M$ . Suppose that after $l - 1 \ge 1$ layers every window's set is an interval of length $(l-1)M$ . Windows of one partition are translates of each other by multiples of $M$ , so the sets of two adjacent windows are intervals of length $(l-1)M$ offset by $M$ . A layer- $l$ window overlaps exactly two adjacent windows of layer $l-1$ , so its set is the union of two such intervals. Since $(l-1)M \ge M$ , the two intervals overlap or touch, and their union is one interval of length $(l-1)M + M = lM$ .

Without the shift, each window overlaps only itself in the previous layer, so the set never grows. $\blacksquare$

In plain words, each layer adds one window width to the receptive field along each axis. That is the same linear growth with depth that §12.6 proved for 1-D sliding windows and that §0.8 computed for stacked convolutions. Patch merging, later in this section, then doubles every receptive field measured in pixels at each stage boundary, which is how Swin's last stage sees the whole image. The cell walks the dependency set backward through the layers on a 56 by 56 map with $M = 7$ .

Python · cell 4

The widget shows the same thing on a small map. Click a token and watch the shaded set it depends on grow, one window per layer.

The cyclic shift, and the mask that makes it exact

The shifted windows are ragged at the borders. Take an 8 by 8 map with $M = 4$ and $\delta = 2$ . Along each axis the shifted windows cover rows 0 and 1, then rows 2 to 5, then rows 6 and 7. That makes 3 by 3 = 9 windows instead of 4, and only the middle one is a full 4 by 4. In general an $H_p\times W_p$ map has $(H_p/M + 1)(W_p/M + 1)$ shifted windows instead of $(H_p/M)(W_p/M)$ , and the border ones are smaller than $M\times M$ .

You could pad every small window to full size and mask the padding. In this example that is 9 windows of work instead of 4, 2.25 times as much. On a large map the extra row and column of windows cost less, but the work is still wasted, and the shifted layers then have a different number of windows from the regular ones.

Swin's trick is to move the image instead of the windows. Roll the map cyclically by $\delta$ toward the top left, so that whatever falls off the top edge comes back at the bottom, and the same for columns. This wrap-around shift is the circular translation that §27.3 used to state translation equivariance exactly. Then cut the rolled map into the regular windows, attend with a mask, and roll the result back.

See it on one axis first. Rolling the rows $0, 1, \dots, 7$ by $-2$ puts them in the order $2, 3, 4, 5, 6, 7, 0, 1$ . The first regular window of the rolled map holds rows 2 to 5, exactly the full shifted window. The second holds rows 6, 7, 0, 1, which glues the bottom piece (rows 6 and 7) to the top piece (rows 0 and 1). Those two pieces were never neighbors in the original map, so a mask must keep them from attending to each other. The example below does this in 2-D by hand.

✏️ The nine regions of an 8 by 8 map, by hand

Take $H_p = W_p = 8$ , $M = 4$ , $\delta = 2$ . Label each row by its shifted window, $g(r) = \lfloor(r + 2)/4\rfloor$ , which is $0$ for rows 0 and 1, $1$ for rows 2 to 5, and $2$ for rows 6 and 7. Label columns the same way, and number region $(g(r), g(c))$ as $3g(r) + g(c)$ , which gives nine regions. After the roll, original rows 2 to 7 come first and rows 0 and 1 wrap to the bottom, and the same for columns. So the label grid of the rolled map is

\begin{bmatrix} 4&4&4&4&5&5&3&3\\ 4&4&4&4&5&5&3&3\\ 4&4&4&4&5&5&3&3\\ 4&4&4&4&5&5&3&3\\ 7&7&7&7&8&8&6&6\\ 7&7&7&7&8&8&6&6\\ 1&1&1&1&2&2&0&0\\ 1&1&1&1&2&2&0&0 \end{bmatrix}.

The top-left rolled window is all region 4, the full shifted window in the middle of the original map, so it needs no mask. The top-right window holds regions 5 and 3, the bottom-left holds 7 and 1, and the bottom-right holds four regions. List the bottom-right window's tokens in raster order and their labels are $(8, 8, 6, 6, 8, 8, 6, 6, 2, 2, 0, 0, 2, 2, 0, 0)$ . Its first two mask rows belong to two tokens of region 8. Writing 1 for a blocked pair ( $-\infty$ in the scores) and 0 for an allowed one, each is

(0,0,1,1,0,0,1,1,1,1,1,1,1,1,1,1).

Each region-8 token attends to the four tokens of its own region, the 2 by 2 shifted window in the bottom-right corner of the original map, and to nothing that wrapped around.

The widget runs the roll on the same map. Press the button, then click a token to see which keys its mask keeps.

The cell checks the hand computation, reusing the row labels of the receptive-field cell.

Python · cell 5

The last line shows the price of the trick. In this tiny map the bottom-right window blocks 75% of its score entries, which are computed and then thrown away. On Swin-T's 56 by 56 first stage only the last row and the last column of windows need a mask at all, 15 of the 64 windows. Here is the construction in general, and the proof that it is exact.

📐 Shifted-window labels and the cyclic-shift mask

Fix $0 \lt \delta \lt M$ . Give original row $r \in \{0, \dots, H_p - 1\}$ the shifted-window index $g(r) = \lfloor (r + M - \delta)/M\rfloor$ , so $g = 0$ for $r \lt \delta$ , then $1, 2, \dots$ , and define $g(c)$ for column $c$ the same way. Token $(r, c)$ belongs to shifted window $(g(r), g(c))$ . Roll the map so that token $(r, c)$ moves to $(\rho, \kappa) = ((r - \delta) \bmod H_p,\ (c - \delta) \bmod W_p)$ , and partition the rolled map into regular windows. Inside each rolled window, the cyclic-shift mask $\mathbf{M}^{\text{sh}} \in \R^{M^2\times M^2}$ is

(\mathbf{M}^{\text{sh}})_{ij} = \begin{cases} 0 & \text{if tokens } i \text{ and } j \text{ have the same label } (g(r), g(c)), \\ -\infty & \text{otherwise.}\end{cases}

Mind the notation. The bold $\mathbf{M}^{\text{sh}}$ is the mask, and the plain $M$ is always the window side.

With the labels in hand, the whole layer is three steps, roll, attend with the mask, and roll back. The claim is that these steps compute exactly what attention over the ragged shifted windows would, border pieces and position biases included, with nothing approximated.

∂ Proof that the cyclic shift is exact

Look at rows. Columns work the same way. Rolled row $\rho$ lies in rolled window $k = \lfloor\rho/M\rfloor$ , and $\rho = (r - \delta) \bmod H_p$ .

For $k \lt H_p/M - 1$ , rolled window $k$ holds rows $\rho \in [kM, kM + M)$ . These are original rows $r \in [kM + \delta, kM + M + \delta)$ , with no wrap-around, and that is exactly shifted window $g = k + 1$ .

The last rolled window, $\rho \in [H_p - M, H_p)$ , holds original rows $r \in [H_p - M + \delta, H_p)$ , the cut-short bottom window, together with the wrapped rows $r \in [0, \delta)$ , the cut-short top window.

So every rolled window is either one whole shifted window or a union of up to four cut-short ones (two pieces per axis), and the mask keeps exactly the pairs inside the same piece. Attention with a mask confined to one piece is attention over that piece, since blocked keys get weight $e^{-\infty} = 0$ and the softmax renormalizes over the rest.

Within one piece every token was moved by the same amount, either $-\delta$ or $-\delta + H_p$ . So row differences, and with them the relative position bias, are the same in rolled and original coordinates. Masked pairs may have wrong offsets, but they get zero weight. Rolling back returns every output to its token. $\blacksquare$

In plain words, the roll glues the ragged border pieces into full-size windows, and the mask keeps the glued pieces from talking to each other. It is also fast. In Swin's speed test on a V100 GPU, the cyclic version ran Swin-T at 755 images per second, against 670 with padding. A naive sliding-window implementation managed 183, and a custom sliding-window kernel 488. The sliding window was about as accurate, 0.1 points higher on ImageNet and 0.3 lower in COCO box AP and ADE20K mIoU (Liu et al. 2021, Tables 5 and 6).

Interviewers like this question because the answer has three parts, roll, mask, and roll back, and each part has a classic bug. The next exercise builds the mask, the middle part.

Interview

The shifted-window attention mask

Implement shifted_window_mask(H, W, M, shift). The token map is H by W, the window side M divides both, and the shift satisfies 0 <= shift < M. Return a float array of shape (n_w, M * M, M * M), one additive mask for each window of the map after it is rolled by (-shift, -shift). Order the windows and the tokens as in Exercise 27.5a.

Entry [k, i, j] is 0 when tokens i and j of rolled window k belong to the same shifted window of the original map, and -100.0 otherwise. Swin's code uses -100 in place of $-\infty$ , because after the softmax a blocked weight is then about $e^{-100} \approx 10^{-44}$ , zero for every practical purpose. With shift = 0 every entry is 0.

import numpy as np

def shifted_window_mask(H, W, M, shift):
    n_w = (H // M) * (W // M)
    return np.zeros((n_w, M * M, M * M))

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Relative position bias in a window

Attention by itself does not know where tokens are. Swin tells it by adding a learned number to each score, chosen by the offset between the query and the key. One number might mean "the key is two rows up and one column left of the query". It adds no absolute position embedding to the tokens at all.

§5.4 met the 1-D version for text, in which each head adds one learned number per distance to its scores, as in T5 (Raffel et al. 2020). There, large distances had to share buckets, because offsets in a sentence are unbounded. Inside a window they are not. Take $M = 2$ , a window of four tokens at $(0,0), (0,1), (1,0), (1,1)$ . Each axis offset is $-1$ , $0$ , or $1$ , so there are only $3\times3 = 9$ possible 2-D offsets, and a 3 by 3 table holds one learned number for each. In general an $M\times M$ window has $2M - 1$ offsets per axis, so Swin can afford one parameter for every 2-D offset.

📐 Relative position bias of a window

Each head $j$ has a table $\hat{\mathbf{B}}_j \in \R^{(2M-1)\times(2M-1)}$ . Tokens $i$ and $i'$ of a window sit at window coordinates $(r_i, c_i)$ and $(r_{i'}, c_{i'})$ . Let $\Delta r = r_i - r_{i'}$ and $\Delta c = c_i - c_{i'}$ (query minus key). The bias added to the score of query $i$ and key $i'$ is

(\mathbf{B}_j)_{ii'} = \hat{\mathbf{B}}_j[\Delta r + M - 1,\ \Delta c + M - 1], \qquad \mathbf{B}_j \in \R^{M^2\times M^2}.

In code the table is stored flat, with $(2M-1)^2$ entries, and read at the index $(\Delta r + M - 1)(2M - 1) + (\Delta c + M - 1)$ . That index is precomputed once as an $M^2\times M^2$ integer matrix.

In plain words, the bias is a learned score for each relative placement of key and query. It is the same for every window and every image, so it is translation equivariant within a window, the property that §27.3 showed a ViT with absolute positions lacks. For $M = 7$ the table has $13^2 = 169$ entries per head, so a three-head block in Swin-T's first stage spends 507 parameters on it. Since the bias depends only on differences of coordinates, $\mathbf{B}_j$ is constant along every diagonal of equal offsets. It is a block-Toeplitz matrix with Toeplitz blocks.

The cell builds the index for $M = 2$ , small enough to check every entry by hand, and confirms for $M = 7$ that equal offsets get equal biases.

Python · cell 6

For $M = 2$ the center of the 3 by 3 table, index 4, is the zero offset, which sits on the diagonal. Row 0 reads 4, 3, 1, 0. The query is at $(0,0)$ and every key is right of it or below it, so $\Delta r$ and $\Delta c$ are 0 or $-1$ , and the indices count down from the center.

Core

Swin's relative position index

Implement relative_position_index(M). It returns an integer array of shape (M * M, M * M) whose entry [i, j] is $(\Delta r + M - 1)(2M - 1) + (\Delta c + M - 1)$ . Token i sits at window coordinates divmod(i, M), and $(\Delta r, \Delta c)$ is the coordinate of token i minus that of token j (query minus key). Use no loops over tokens.

import numpy as np

def relative_position_index(M):
    rc = np.stack(np.divmod(np.arange(M * M), M), axis=1)
    d = rc[:, None, :] - rc[None, :, :] + (M - 1)
    return d[..., 0] * M + d[..., 1]

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

How much does the bias matter? Swin-T with it reaches 81.3% top-1, 50.5 box AP, and 46.1 mIoU. With no position information at all the numbers are 80.1%, 49.2, and 43.8. With ViT's absolute embedding instead they are 80.5%, 49.0, and 43.2. So the absolute embedding helps classification a little and hurts both dense tasks. Adding it on top of the relative bias leaves top-1 at 81.3% but costs 0.3 box AP and 2.1 mIoU, which is why Swin uses the bias alone (Liu et al. 2021, Table 4). Dense tasks care more about translation equivariance than classification does.

Indexing the table by offset has a practical consequence. Fine-tuning at a larger resolution usually means a larger window. The released Swin-B at 384 pixels uses $M = 12$ , so its 13 by 13 table is resized to 23 by 23 by bicubic interpolation, the same move as resizing a ViT's position table in §27.1. Stretching a table far beyond its trained range works badly, and Swin V2 fixed that. In an interview, "how does Swin encode position, and what breaks when the window grows?" has exactly this answer.

🔬 Swin V2, for larger windows and larger models

Swin V2 (Liu et al. 2022a) made three changes so that windows could grow and models could scale.

First, it generates the bias with a small MLP of the log-spaced offsets $\widehat{\Delta x} = \operatorname{sign}(\Delta x)\ln(1 + |\Delta x|)$ , instead of reading a table. Take a model trained with 8 by 8 windows (offsets up to 7) and run it with 16 by 16 windows (offsets up to 15). In linear coordinates the new offsets reach $(15 - 7)/7 = 1.14$ times past the trained range. In log coordinates they reach only $(\ln 16 - \ln 8)/\ln 8 = 0.33$ times past it.

Second, it replaced the dot product with a cosine similarity divided by a learned temperature $\tau \gt 0.01$ , which keeps a few pixel pairs from dominating an attention map.

Third, it moved each LayerNorm to the end of its residual branch ("res-post-norm"), which keeps activation sizes from accumulating layer by layer. The cosine and the norm address two faces of the training instability that §5.7 discusses for language models.

With these changes and masked-image pretraining (Xie et al. 2022), a 3-billion-parameter SwinV2-G trained at up to 1,536 by 1,536 pixels reached 63.1 box AP on COCO test-dev and 59.9 mIoU on ADE20K. Both numbers use multi-scale testing, and the COCO one also used extra detection data (Objects365) before fine-tuning.

Patch merging, and Swin counted to the last parameter

Windows make the cost linear. The pyramid comes from merging neighboring tokens between stages, which halves the map and doubles the channels.

In plain words, every 2 by 2 block of tokens becomes one token with twice the channels. Concatenating a 2 by 2 block and applying a linear map is the patch embedding of §27.1 applied to tokens instead of pixels. So patch merging is a LayerNorm over each 2 by 2 block followed by a 2 by 2 convolution with stride 2 and no bias.

Now the whole network. Swin first cuts the image into 4 by 4 pixel patches of 48 numbers each (16 pixels times 3 colors). It projects them to width $d_1$ , exactly a ViT patch embedding with $P = 4$ . Swin calls this step "patch partition". Four stages follow, separated by three patch merges. Every head has $d_k = 32$ channels, and the MLP expands the width 4 times. The table counts all four sizes exactly, with a 1,000-class head at 224 pixels and $M = 7$ . The counts include every weight and bias and match the timm library's, and they sit next to the paper's numbers.

model	$d_1$	blocks per stage	heads per stage	parameters (exact)	multiply-adds	ImageNet-1k top-1
Swin-T	96	2, 2, 6, 2	3, 6, 12, 24	28,288,354	4.5G	81.3%
Swin-S	96	2, 2, 18, 2	3, 6, 12, 24	49,606,258	8.7G	83.0%
Swin-B	128	2, 2, 18, 2	4, 8, 16, 32	87,768,224	15.4G	83.5% (84.5% at 384)
Swin-L	192	2, 2, 18, 2	6, 12, 24, 48	196,532,476	34.5G	87.3% at 384 (103.9G), ImageNet-21k pretraining

The accuracies are from Liu et al. (2021), Table 1. Every model but Swin-L was trained on ImageNet-1k alone (1.28 million images in 1,000 classes). The recipe was the one of §27.4, 300 epochs of AdamW with strong augmentation, Mixup and CutMix, and stochastic depth. Stochastic depth skips whole residual branches at random during training, here at rates 0.2, 0.3, and 0.5 for T, S, and B. That recipe was introduced by DeiT (Touvron et al. 2021). Swin-L was pretrained on ImageNet-21k, the full ImageNet of 14.2 million images in about 21,800 classes, which the Swin paper calls ImageNet-22K. The paper rounds the parameter counts to 29M, 50M, 88M, and 197M. The ConvNeXt paper lists Swin-T as 28M, so even the rounding varies by one, which is a good reason to count.

✏️ Swin-T, every parameter and multiply-add

A Swin block at width $d$ with $h$ heads has two LayerNorms ( $4d$ ), the QKV projection with bias ( $3d^2 + 3d$ ), the output projection ( $d^2 + d$ ), the bias table ( $169h$ ), and the MLP ( $4d^2 + 4d + 4d^2 + d$ ). That is $12d^2 + 13d + 169h$ parameters, the Transformer block of §5.6 plus the bias tables. For Swin-T,

the patch embedding has $48\cdot96 + 96$ parameters plus $192$ for its LayerNorm, which is $4{,}896$ ;
stage 1 has two blocks at $d = 96$ , $h = 3$ , so $2(110{,}592 + 1{,}248 + 507) = 224{,}694$ ;
the merges into stages 2, 3, and 4 cost $8d^2 + 8d$ for $d = 96, 192, 384$ , which is $74{,}496 + 296{,}448 + 1{,}182{,}720 = 1{,}553{,}664$ ;
stages 2, 3, and 4 have blocks at $d = 192, 384, 768$ with $h = 6, 12, 24$ , so $2\cdot445{,}878 + 6\cdot1{,}776{,}492 + 2\cdot7{,}091{,}928 = 25{,}734{,}564$ ;
the final LayerNorm and the classifier add $1{,}536 + 768{,}000 + 1{,}000 = 770{,}536$ .

The total is 28,288,354. For the multiply-adds at 224 pixels, each block of stage $s$ costs $12T_sd_s^2 + 2M^2T_sd_s$ with $T_s = H_sW_s$ . Since $T_sd_s^2 = 3136\cdot96^2 \approx 28.9$ M at every stage, the linear layers cost $12\cdot28.9\text{M} \approx 347$ M per block in every stage, as the pyramid argument promised. Twelve blocks give 4.16G. The attention scores add only 0.14G, the merges 0.17G, and the stem and head 0.015G, for 4.49G. Only about 3% of Swin-T's compute is the attention scores themselves.

The cell repeats the count for all four sizes.

Python · cell 7

Core

The cost of a Swin backbone

First implement attention_macs(H_p, W_p, d, M=None), the multiply-adds of one attention layer from the cost theorem, projections included. It is global attention when M is None and window attention otherwise.

Then implement swin_macs(img, P, d1, depths, M=7, mlp_ratio=4, n_classes=1000) for a square image of side img. It returns a dict with the keys "embed", "attn", "mlp", "merge", "head", "total", and "stages". Count only matrix multiplies, with these rules.

The patch embedding costs $H_1W_1P^2\cdot3\cdot d_1$ .
Each block of stage $s$ adds attention_macs at that stage to "attn" and $2\cdot\text{mlp\_ratio}\cdot T_sd_s^2$ to "mlp". The window side is $\min(M, H_s, W_s)$ , because Swin shrinks the window when the map is smaller than it.
Each merge costs $T_{s+1}\cdot4d_s\cdot2d_s$ , and the head costs $d_S\cdot n_{\text{classes}}$ .
"stages" is the list of $(H_s, W_s, d_s)$ .

Use Python integers.

def attention_macs(H_p, W_p, d, M=None):
    T = H_p * W_p
    return 2 * T * T * d if M is None else 2 * M * M * T * d

def swin_macs(img, P, d1, depths, M=7, mlp_ratio=4, n_classes=1000):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

A Swin block in PyTorch

Production code does exactly what we derived. The block below runs one shifted-window layer with torch.roll, the region mask, and the relative bias. It then compares the result with a slow reference that gathers each true shifted window of the original map and attends inside it. Agreement is the cyclic-shift theorem checked numerically.

🔥 In PyTorchfor a machine with PyTorch

Interview

Shifted window attention, end to end

Put the pieces together. Implement shifted_window_attention(x, Wqkv, Wo, table, M, shift, h) for one token map x of shape (H, W, d), and return an array of shape (H, W, d). The arguments follow these conventions.

Wqkv has shape (d, 3d). Its first d columns make the queries, the next d the keys, and the last d the values. Each block of d columns splits into h heads of d // h consecutive columns.
Wo has shape (d, d), and there are no biases.
table has shape (h, (2M - 1)**2) and is read with the index of Exercise 27.5c.
Scale the scores by $1/\sqrt{d_k}$ , with $d_k = d/h$ .
If shift > 0, roll by (-shift, -shift), add the mask of Exercise 27.5b, and roll the output back.

The setup provides working copies of the earlier exercises, ws_partition(x, M) (for (H, W, d) maps), ws_reverse(w, M, H, W), ws_rel_index(M), and ws_mask(H, W, M, shift). The tests compare your output with attention computed directly inside each shifted window of the original map, with no roll at all.

import numpy as np

def shifted_window_attention(x, Wqkv, Wo, table, M, shift, h):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Swin delivered what it promised. The results below come from standard detection and segmentation heads that §27.6 explains (Cascade Mask R-CNN and HTC++ on COCO, UperNet on ADE20K), with the backbone as the only thing changed. With ImageNet-21k pretraining, Swin-L reached 58.7 box AP and 51.1 mask AP on COCO test-dev and 53.5 mIoU on ADE20K, all three with multi-scale testing, the best results of early 2021. At similar size and FLOPs, Swin-T beat a ResNet-50 on COCO by 4.2 box AP, 50.5 against 46.3 (Liu et al. 2021, Tables 2 and 3).

ConvNeXt rebuilds a ResNet with Transformer-era choices

Swin's results raise a question. Was it attention that won, or everything that came with it? Compared with a ResNet, Swin changed many things at once. It used 4 by 4 patches, a different share of blocks per stage, a wide MLP, LayerNorm, the GELU activation, and fewer nonlinearities. It was also trained with the long, heavily regularized recipe described above.

Liu et al. (2022b) tested this the way a careful experimenter should. They started from ResNet-50 and changed one thing at a time toward Swin-T. They kept the network a pure convolutional network and the compute near 4.5G multiply-adds, and trained every step with three random seeds. The result is called ConvNeXt.

📌 Modernizing a ResNet-50, step by step (ConvNeXt, Table 10)

step	ImageNet-1k top-1	GFLOPs
ResNet-50, original training (torchvision)	76.13%	4.09
the DeiT and Swin training recipe, 300 epochs	78.82%	4.09
stage ratio $(3, 4, 6, 3) \to (3, 3, 9, 3)$	79.36%	4.53
"patchify" stem, a 4 by 4 convolution with stride 4	79.51%	4.42
depthwise 3 by 3 convolution (a grouped convolution with one group per channel)	78.28%	2.35
width 64 to 96, as in Swin-T	80.50%	5.27
inverted bottleneck (expand 4 times, as in a Transformer MLP)	80.64%	4.64
move the depthwise convolution to the top of the block	79.92%	4.07
depthwise kernel 3 to 7 (5 gives 80.35%, 9 gives 80.57%, 11 gives 80.47%)	80.57%	4.15
ReLU to GELU	80.62%	4.15
one activation per block instead of three	81.27%	4.15
one normalization per block	81.41%	4.15
BatchNorm to LayerNorm	81.47%	4.46
separate 2 by 2, stride-2 downsampling layers with LayerNorm (ConvNeXt-T)	81.97%	4.49
Swin-T, for comparison	81.30%	4.50

The "GFLOPs" here are billions of multiply-adds, as everywhere in vision papers (§27.2). Each accuracy is a mean over three seeds, and the paper's main table lists ConvNeXt-T at 82.1%.

Read the table as an argument. The largest single gain, 2.7 points, is the training recipe of §27.4, before any change to the architecture. Then each choice borrowed from Transformers maps to a convolutional one.

Patchify stem. A 4 by 4 convolution with stride 4 is the patch embedding of §27.1 with $P = 4$ . It replaces the ResNet stem's overlapping 7 by 7 stride-2 convolution and max pooling (§0.8).
Depthwise convolution. A depthwise convolution filters each channel on its own, so it mixes information only across space. A 1 by 1 convolution mixes only across channels. That is the same split as attention (space) and the MLP (channels) in a Transformer block.
Inverted bottleneck. The block widens to 4 times its channels in the middle and narrows back, like the Transformer MLP. A ResNet bottleneck does the opposite.
Depthwise convolution moved to the top. This mirrors putting attention before the MLP, and it lets the spatial operation run on $d$ channels instead of $4d$ .
7 by 7 kernel. This is the size of a Swin window. Larger kernels gave no further gain.
Fewer activations and norms. One GELU, one LayerNorm, and separate downsampling layers with their own norm copy the Transformer block's sparing use of nonlinearities and norms. BatchNorm, which normalizes each channel with statistics of the batch (§0.7), gives way to LayerNorm, which normalizes each position over its channels.

📐 ConvNeXt block

For a feature map $\mathbf{x} \in \R^{H_s\times W_s\times d}$ , a ConvNeXt block computes

\mathbf{y} = \mathbf{x} + \boldsymbol\gamma^{\text{LS}}\odot\operatorname{PW}_2\Big(\operatorname{GELU}\big(\operatorname{PW}_1(\operatorname{LN}(\operatorname{DWConv}_{7\times7}(\mathbf{x})))\big)\Big),

where

$\operatorname{DWConv}_{7\times7}$ is a depthwise convolution with weights $\mathbf{W}_{\text{dw}} \in \R^{7\times7\times d}$ , a bias, stride 1, and zero padding 3, so each channel is filtered separately and the map keeps its size;
LN normalizes each position over its $d$ channels, with $\epsilon = 10^{-6}$ ;
$\operatorname{PW}_1(\mathbf{u}) = \mathbf{u}\mathbf{W}_1 + \mathbf{b}_1$ with $\mathbf{W}_1 \in \R^{d\times4d}$ and $\operatorname{PW}_2(\mathbf{u}) = \mathbf{u}\mathbf{W}_2 + \mathbf{b}_2$ with $\mathbf{W}_2 \in \R^{4d\times d}$ act on each position separately (1 by 1 convolutions);
GELU is the exact form, $\tfrac12u(1 + \operatorname{erf}(u/\sqrt2))$ ;
$\boldsymbol\gamma^{\text{LS}} \in \R^d$ is LayerScale, a learned gain per channel initialized to $10^{-6}$ (§27.4).

The block has $8d^2 + 58d$ parameters and costs $T_s(49d + 8d^2)$ multiply-adds on $T_s = H_sW_s$ positions.

In plain words, filter each channel over a 7 by 7 neighborhood, normalize, run a Transformer MLP at every position, scale the result by a tiny learned gain, and add it back. Count the parameters to see where the $58d$ comes from. The depthwise weights and bias are $49d + d$ , the LayerNorm $2d$ , the two MLP biases $4d + d$ , and LayerScale $d$ , which sums to $58d$ , next to the $8d^2$ of the two matrices. Because LayerScale starts at $10^{-6}$ , the residual branch starts scaled almost to zero. Every block therefore begins as nearly the identity, the same reason LayerScale helped deep ViTs train in §27.4.

A short count checks the claim that ConvNeXt-T and Swin-T have the same compute. Per position, a Swin block costs $12d^2$ plus a little attention, and a ConvNeXt block costs $8d^2$ plus $49d$ . Swin-T has 2, 2, 6, 2 blocks per stage and ConvNeXt-T has 3, 3, 9, 3. In units of $T_sd_s^2$ , the stages cost $(24, 24, 72, 24)$ in both networks. The extra depth of ConvNeXt pays exactly for its cheaper blocks.

Interview

A ConvNeXt block in NumPy

Implement convnext_block(x, params) for x of shape (H, W, d), exactly as the definition says. The dict params holds these arrays.

"dw" of shape (7, 7, d), applied as a cross-correlation, which is the PyTorch convention. The output at $(r, c)$ is $\sum_{a,b}$ dw[a, b] * x[r + a - 3, c + b - 3] for each channel, with zeros outside the map.
"dw_b", "ln_g", and "ln_b", each of shape (d,). The LayerNorm runs over channels with $\epsilon = 10^{-6}$ and the biased variance.
"W1" of shape (d, 4d), "b1", "W2" of shape (4d, d), "b2", and "gamma" of shape (d,).

Use the exact GELU, $\tfrac12u(1 + \operatorname{erf}(u/\sqrt2))$ , with scipy.special.erf. Use no loops over positions or channels. A loop over the 49 kernel taps is fine.

import numpy as np

def convnext_block(x, params):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In PyTorch the depthwise convolution is nn.Conv2d(d, d, 7, padding=3, groups=d). The official code permutes the map to channels-last, so that LayerNorm and the two linear layers act on the last axis. The block below builds the layer and checks it against a NumPy version written inside the same block.

🔥 In PyTorchfor a machine with PyTorch

nn.Linear stores its weight as (output, input), the transpose of this chapter's $\mathbf{W}_1$ , which is why the NumPy check multiplies by .T.

The result held beyond ImageNet. ConvNeXt-T reached 82.1% at 4.5G against Swin-T's 81.3%, and ConvNeXt-B reached 83.8% against Swin-B's 83.5%. With ImageNet-21k pretraining, ConvNeXt-XL reached 87.8% at 384 pixels. On COCO with Cascade Mask R-CNN, ConvNeXt-T and Swin-T tie at 50.4 box AP, and with ImageNet-21k pretraining ConvNeXt-B leads Swin-B by 1.0 box AP, 54.0 against 53.0. On ADE20K with UperNet, ConvNeXt is ahead at every size by 0.1 to 1.4 mIoU (Liu et al. 2022b, Tables 1, 3, and 4).

The paper's own conclusion is the one to remember. Both networks beat a plain ViT on efficiency because they compute locally, an inductive bias, and not because of self-attention as such. §27.3 reached the same lesson, where carefully trained convolutional networks matched ViTs at equal compute. If an interviewer asks "is attention necessary for a strong vision backbone?", ConvNeXt is the answer to cite, with its recipe gain of 2.7 points as the headline.

🔬 ConvNeXt V2 and global response normalization

ConvNeXt V2 (Woo et al. 2023) adapted the design to masked-image pretraining, which §28.5 teaches. Masked autoencoding with a convolutional network first suffered from feature collapse in the MLP, with many channels dead or saturated. A new layer called global response normalization (GRN) fixed it.

For a map $\mathbf{X} \in \R^{H\times W\times C}$ , GRN computes each channel's spatial L2 norm $g_i = \lVert\mathbf{X}_{:,:,i}\rVert_2$ , divides it by the mean over channels, $n_i = g_i/(\frac1C\sum_j g_j)$ , and returns $\gamma_i\mathbf{X}_{:,:,i}n_i + \beta_i + \mathbf{X}_{:,:,i}$ , with $\gamma$ and $\beta$ initialized to zero. Channels with a large response are boosted relative to the rest, so channels compete. GRN made LayerScale unnecessary. The family runs from a 3.7M-parameter Atto model at 76.7% top-1 to a 650M-parameter Huge model at 88.9%, using only public data.

Hybrids put convolutions early and attention late

Convolutions suit the early layers of a network, where features are local and maps are large, because they are cheap and need less data. Attention suits the late layers, where the map is small and mixing that depends on the input pays for itself. Many designs use both on purpose. Here are the three ideas worth knowing.

Convolutional stems. Xiao et al. (2021) replaced ViT's 16 by 16 patchify stem with about five convolutions. These are a stack of 3 by 3 convolutions with BatchNorm and ReLU, in the smaller models four of them with stride 2 to reach stride 16, and a final 1 by 1 convolution to width $d$ . The stem costs about as much as one Transformer block, which they removed to keep the FLOPs equal. The ViT became far less sensitive to the choice of optimizer, its learning rate, and the schedule length, and gained about 1 to 2 points of ImageNet top-1. The authors' explanation is that a first layer with a large kernel and a large stride, unusual for a convolution, makes optimization harder.

Relative attention. Dai et al. (2021) observed that depthwise convolution and attention are both weighted sums of other positions. A depthwise convolution computes $\mathbf{y}_i = \sum_{j} w_{i-j}\mathbf{x}_j$ , with a fixed weight per offset over a local window. Attention computes $\mathbf{y}_i = \sum_j A_{ij}\mathbf{x}_j$ , with weights that depend on the input, over everything. Adding a learned scalar $w_{i-j}$ to the attention scores before the softmax combines the two. The offset term gives translation equivariance, and the dot product gives weights that adapt to the input. This is the same relative bias Swin uses, applied globally. Their network, CoAtNet, stacks two convolutional stages and then two Transformer stages (C-C-T-T). It reached 86.0% on ImageNet-1k alone and 88.56% with ImageNet-21k pretraining, which matched a ViT-H/14 pretrained on 300 million images (JFT-300M, a Google-internal labeled set) with 23 times less data. With 3 billion images (JFT-3B) it reached 90.88%.

Spatial-reduction attention. The Pyramid Vision Transformer (Wang et al. 2021) kept global attention in a four-stage pyramid. To make it affordable, it shrinks the keys and values by a factor $R$ per side before attending, so the score term drops from $2T^2d$ to $2T^2d/R^2$ . Each query still sees the whole image, but through a coarser grid of keys.

On phones and embedded chips the target is measured latency, the time per image on the device, and the winning designs mix cheap convolutions with a little attention. The box below lists the main ones, and the far end of the spectrum, a network with no attention at all.

🔬 Mobile hybrids, and the all-MLP extreme

LeViT (Graham et al. 2021) combined a convolutional stem, shrinking stages, and a learned attention bias. At 80% ImageNet top-1 it ran 5 times faster than EfficientNet on a CPU.
MobileViT (Mehta and Rastegari 2022) put small Transformer blocks between MobileNet blocks and reached 78.4% with about 6M parameters.
FastViT (Vasu et al. 2023) trains with extra branches and skip connections that are folded into single convolutions for inference, a technique called structural reparameterization. It ran 1.9 times faster than ConvNeXt on a phone at equal accuracy.
Two different models are called EfficientViT. Cai et al. (2023) use multi-scale ReLU linear attention for high-resolution segmentation. Liu et al. (2023) found that the reshapes and elementwise operations inside attention, not its FLOPs, limited the speed of small ViTs.

MLP-Mixer (Tolstikhin et al. 2021) removed attention too. A Mixer layer applies one MLP across the tokens (the same for every channel) and one across the channels (the same for every token). With JFT-300M pretraining, Mixer-H/14 reached 87.94% on ImageNet, 0.6 points below ViT-H/14, and the paper reports it running 2.5 times faster. But its token-mixing MLP has a fixed input size, the number of patches, so it is tied to one resolution, the opposite of what dense prediction needs. It shows how far fewer priors can go with enough data, the lesson of §27.3 again.

The table puts the designs side by side. The cost column is how the token-mixing step grows with the token count $T$ .

design	token mixing	cost in tokens $T$	changes resolution freely	example result
plain ViT	global attention	$T^2$	needs position interpolation	ViT-B/16 on IN-1k alone, 77.9% at 384 (2020 recipe), 81.8% at 224 (DeiT)
Swin	window attention plus shift	$T$	relative bias, window may need resizing	Swin-T 81.3%, 4.5G
ConvNeXt	7 by 7 depthwise convolution	$T$	yes, fully convolutional	ConvNeXt-T 82.1%, 4.5G
CoAtNet	convolution early, relative attention late	$T$ early, $T^2$ late on small maps	relative bias	86.0% IN-1k alone
PVT	attention with reduced keys and values	$T^2/R^2$	position interpolation	PVT-Small, 40.4 AP with RetinaNet (1x schedule) on COCO val
MLP-Mixer	MLP across tokens	$T^2$ (a $T\times T$ weight)	no, fixed $T$	87.94% with JFT-300M

Plain ViTs return with self-supervised pretraining

By 2022 the story seemed settled, with hierarchical backbones for dense tasks and plain ViTs for classification. Then self-supervised pretraining, which learns from images without labels, changed the economics. The method that mattered most is the masked autoencoder, MAE (He et al. 2022), which §28.5 develops. MAE hides 75% of the patches and trains the ViT to reconstruct them from the rest.

That is natural for a plain ViT, which reads any set of tokens, and awkward for windows and convolutions, which want a full grid. If pretraining can teach a plain ViT locality and scale, the backbone may not need them built in.

ViTDet (Li et al. 2022a) tested this for detection. It took an MAE-pretrained plain ViT and changed it only during fine-tuning, in three ways.

Most blocks use window attention without any shift, with windows the size of the pretraining grid (14 by 14 tokens).
Four blocks, spread evenly through the network, use global attention so that information crosses the windows.
Instead of a feature pyramid network, a simple feature pyramid is built from the last stride-16 map alone. A 2 by 2 max pooling of stride 2 gives stride 32, and one or two 2 by 2 transposed convolutions (learned upsampling layers) give strides 8 and 4.

With ViT-H and MAE pretraining on ImageNet-1k alone, ViTDet with a Cascade Mask R-CNN head reached 60.4 box AP on COCO with single-scale testing and 61.3 with multi-scale testing. That beat every hierarchical-backbone result in its comparison, including those pretrained on ImageNet-21k with labels.

The same recipe now runs inside widely used models. SAM, the promptable segmentation model of §27.6, has an image encoder that §27.2 counted. It uses 14 by 14 windows plus four global blocks on a 64 by 64 grid. So does the vision encoder of the Qwen2.5-VL vision-language model (Bai et al. 2025), a 32-block ViT of width 1,280 with 14-pixel patches that reads images at their native resolution. Its blocks 7, 15, 23, and 31 (counting from 0) attend globally. The other 28 attend within windows of at most 112 by 112 pixels, 8 by 8 patches, with 2-D rotary position embeddings (§27.1). An image of any size and shape therefore costs attention linear in its area in all but four blocks, and §31.5 follows its tokens into the language model.

Hiera (Ryali et al. 2023) did the same for hierarchical models. It started from MViTv2 (Li et al. 2022b), a strong hierarchical ViT that shrinks its maps by pooling inside attention. With MAE pretraining it removed MViTv2's convolutions, its relative position embeddings, and its other vision-specific modules one by one, without losing accuracy. What remains is a stack of standard ViT blocks with three changes. The first two stages attend locally inside mask units, the blocks of 8 by 8 tokens (32 by 32 pixels) that MAE hides or keeps as a whole. Later stages attend globally. And the queries are max-pooled 2 by 2 at each stage transition, which halves the map.

Hiera-B reached 84.5% on ImageNet-1k at 9G multiply-adds. That is more accurate than Swin-B at about 60% of its FLOPs, and it ran faster in wall-clock time than earlier hierarchical models. Hiera became the image encoder of SAM 2 (Ravi et al. 2024).

The trend continued with scale. DINOv3 (Siméoni et al. 2025) trained a 6.7B-parameter plain ViT without labels, by the self-distillation that §28.4 teaches. With that backbone frozen, a set-prediction detector in the style of DETR (§27.6) reached 66.1 box AP on COCO. A Mask2Former segmentation head on the same frozen backbone reached 63.0 mIoU on ADE20K. Both numbers use test-time augmentation, and without it they are 65.6 and 62.6. Both were among the strongest published results when they appeared. Its weights were also distilled into smaller ViTs and into ConvNeXt-T, S, B, and L, for devices where a 7B model cannot run.

Real-time detectors show the same split. RF-DETR (Robinson et al. 2026) builds on a DINOv2 ViT with interleaved windowed and global attention. DEIMv2 (Huang et al. 2025) uses DINOv3 ViTs, pretrained or distilled, for its larger models and a pruned convolutional backbone, HGNetv2, for its smallest ones.

So the current answer is neither "hierarchical beats plain" nor the reverse. Priors and pretraining substitute for each other. With little data, build locality and scale into the architecture. With massive self-supervised pretraining, a plain ViT learns them, and its simplicity and scalability win, as long as windows or a few global blocks keep high-resolution fine-tuning affordable. That trade-off is the core of the "plain or hierarchical?" interview question.

Fewer tokens instead of cheaper attention

Everything so far made each token cheaper, either by limiting what it attends to or by replacing attention with a convolution. The other lever keeps a trained plain ViT exactly as it is and gives it fewer tokens. A block costs $12Td^2 + 2T^2d$ , at least linear in $T$ , so every token removed saves at least $12d^2$ multiply-adds in each block it would have passed through. This matters because the strongest available weights are often a plain ViT (from MAE, CLIP, or DINOv2), and retraining one with windows costs more than most teams can spend.

Removing tokens is natural for a plain ViT and awkward for a hierarchical one, for three reasons. Each token has carried its position since the input embedding (§27.1), so it needs no neighbors to say where it is. Attention and the MLP are permutation equivariant (§4.6), meaning that reordering the input tokens only reorders the outputs, so a block runs on any set of tokens in any order. And a classifier reads only the class token. Windows, shifted windows, and convolutions all need the full grid in every layer.

There are three ways to use fewer tokens.

Bigger patches cut the count from the start, by a factor $(P'/P)^2$ for patch side $P'$ instead of $P$ . FlexiViT (§27.1) lets one trained model run at several patch sizes. But a bigger patch coarsens every region equally, the sky as much as a face.
Pruning deletes the tokens that some score calls unimportant.
Merging averages tokens that carry nearly the same information, so a flat sky collapses into a few tokens while a detailed region is left alone.

Token Merging, or ToMe (Bolya et al. 2023), is the standard way to merge, and it works on a trained model with no retraining at all. Here is its idea on seven tokens before the definition.

✏️ One merging step by hand

Take seven tokens, numbered 0 to 6, with token 0 the class token. Split them by position into $A = \{0, 2, 4, 6\}$ and $B = \{1, 3, 5\}$ . Each token of $A$ finds its most similar token of $B$ , measured by the cosine of their attention keys. Suppose token 6's key is almost parallel to token 1's (cosine 0.999), while tokens 2 and 4 point away from every token of $B$ (cosines of at most 0.01). The class token is never allowed to merge.

With $r = 1$ , one merge per block, the most confident proposal wins, so token 6 is averaged into token 1. If token 1 already stood for 3 patches and token 6 for 1, the merged token is $(3\mathbf{x}_1 + 1\mathbf{x}_6)/4$ , and it now stands for 4 patches. Six tokens go on to the MLP and the later blocks. Tokens 2 and 4 might have nearly equal keys, but both are in $A$ , so they cannot merge with each other in this step.

📐 Bipartite soft matching (ToMe)

Let $\mathbf{X} \in \R^{n\times d}$ be the residual stream of one image right after the attention branch of a block, with row 0 the class token. Let $\mathbf{s} \in \R^n$ hold the sizes, the number of input patches each token stands for (all 1 at the input). Let $\bar{\mathbf{K}} \in \R^{n\times d_k}$ be this block's attention keys averaged over the $h$ heads, with rows $\bar{\mathbf{k}}_i$ . Fix an integer $r$ , a number of tokens and not a ratio, with $0 \le r \le \lfloor(n-1)/2\rfloor$ .

Split. Let $A = \{0, 2, 4, \dots\}$ be the even rows and $B = \{1, 3, 5, \dots\}$ the odd rows.
Match. Each $a \in A$ proposes the partner $b(a) = \arg\max_{b\in B}\cos(\bar{\mathbf{k}}_a, \bar{\mathbf{k}}_b)$ , with score $\sigma_a = \cos(\bar{\mathbf{k}}_a, \bar{\mathbf{k}}_{b(a)})$ , except the class token, which gets $\sigma_0 = -\infty$ .
Merge. Let $S \subset A$ be the $r$ tokens with the largest $\sigma_a$ , and for each $b \in B$ let $S_b = \{a \in S : b(a) = b\}$ . Every $b \in B$ becomes

\mathbf{x}_b \leftarrow \frac{s_b\mathbf{x}_b + \sum_{a\in S_b}s_a\mathbf{x}_a}{s_b + \sum_{a\in S_b}s_a}, \qquad s_b \leftarrow s_b + \sum_{a\in S_b}s_a ,

and the tokens of $S$ are deleted, leaving $n - r$ .

The block's MLP and every later block run on the merged set. Their attention is proportional attention, $\operatorname{softmax}\big(\mathbf{Q}\mathbf{K}^\top/\sqrt{d_k} + \mathbf{1}(\ln\mathbf{s})^\top\big)$ , which adds $\ln s_j$ to every score of key $j$ . A model without a class token skips the protection and may take $r$ up to $\lceil n/2\rceil$ .

In plain words, every token of $A$ proposes its closest token of $B$ , and the $r$ most confident proposals are accepted. Each choice has a reason, and ToMe's ablations put a number on each (Bolya et al. 2023, Table 1). They use an MAE ViT-L/16 off the shelf, at 85.96% top-1 before merging, and merge $r = 8$ tokens per block.

Two sets. A destination is never itself merged away, so there are no chains and no clustering. The work is one $|A|\times|B|$ similarity matrix and a sort, which runs nearly as fast as dropping tokens at random.
Alternating positions. Tokens next to each other in raster order land in opposite sets, so horizontally adjacent patches can merge. This scores 84.25%, against 81.07% when $A$ is the first half of the sequence and $B$ the second.
Keys. The keys summarize what attention already uses to compare tokens, while the features carry much else. Keys score 84.25%, the features after attention 83.70%, and those before it 83.02%.
Size weighting. If $\mathbf{x}_a$ and $\mathbf{x}_b$ are averages of the $s_a$ and $s_b$ patch vectors they stand for, the size-weighted average is exactly the average of all $s_a + s_b$ of them. A plain average would let one stray patch count as much as ten merged ones. Weighting scores 84.25%, against 83.57%.

Interview

One step of token merging

Implement tome_merge(X, K, sizes, r), one step of bipartite soft matching on one image, exactly as the definition says. X has shape (n, d) with the class token in row 0, K has shape (n, d_k) (the keys, already averaged over heads), and sizes has shape (n,). Return (X_new, sizes_new) of shapes (n - r, d) and (n - r,), with these conventions.

Keep the surviving tokens in their original order, so the class token stays in row 0. ToMe's own code returns the survivors of $A$ followed by those of $B$ . A Transformer does not mind, since attention is permutation equivariant, but the tests expect the original order.
Break ties by the lower index, with a stable sort.
Use no Python loops over tokens.

This is a favorite inference-efficiency question. Its classic slips are matching within one set, averaging without the sizes, comparing features instead of keys, and merging the class token.

import numpy as np

def tome_merge(X, K, sizes, r):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Why does a merged token need the $\ln s$ term in later attention? Without it, a token that stands for $s$ patches gets the weight of one.

∴ Proportional attention reproduces duplicated tokens

Let a query $\mathbf{q} \in \R^{d_k}$ attend over $m$ distinct tokens with keys $\mathbf{k}_j$ and values $\mathbf{v}_j$ , where token $j$ occurs $s_j$ times, and write $z_j = \mathbf{q}^\top\mathbf{k}_j/\sqrt{d_k}$ . Attention over all $\sum_j s_j$ copies equals proportional attention over the $m$ distinct tokens,

\frac{\sum_{j=1}^m s_je^{z_j}\mathbf{v}_j}{\sum_{j=1}^m s_je^{z_j}} = \sum_{j=1}^m \operatorname{softmax}(\mathbf{z} + \ln\mathbf{s})_j\,\mathbf{v}_j .

So if tokens that are exact copies of each other are merged by size-weighted averaging, and every later block uses proportional attention, every remaining token, the class token included, computes exactly what it would have computed without merging.

∂ Proof of the proportional-attention identity

The $s_j$ copies of token $j$ contribute $s_je^{z_j} = e^{z_j + \ln s_j}$ to both the numerator and the denominator, which is the identity.

For the consequence, copies have equal queries, keys, and values, so by the identity they receive equal attention outputs. The rest of a block (the norms, the MLP, the residual) acts on each token separately, so copies that enter a block leave it as copies. By induction over the blocks they stay copies. Their size-weighted average therefore equals each of them, and by the identity every other token's attention sees the merged token exactly as it saw the copies. $\blacksquare$

In plain words, merging costs nothing when the merged tokens were identical, and little when they were close, as long as the merged token keeps the weight of all its parts. In practice the term matters for some weights and not others. It helped supervised weights, a ViT-L/16 trained with the augmentation and regularization recipe of Steiner et al. (2021), which went from 82.15% to 83.51% with it. It did not help MAE weights used off the shelf (84.25% without, 83.84% with), so ToMe turns it off for those. The cell checks the identity on five distinct tokens with 11 copies in all.

Python · cell 8

How much does merging save? ToMe merges the same $r$ tokens in every block, so every image keeps the same count and batches stay rectangular.

∴ Cost of a constant merging schedule

Let a plain ViT of width $d$ with $L$ blocks read $T$ tokens, one of them a class token, and merge $r$ tokens in every block. Block $\ell = 1, \dots, L$ attends over $n_\ell = T - (\ell - 1)r$ tokens and runs its MLP on $n_\ell - r$ . Every merge respects the bound $r \le \lfloor(n_\ell - 1)/2\rfloor$ if and only if $(L + 1)r \le T - 1$ . The network then ends with $T - rL$ tokens and costs

\Omega_{\text{ToMe}} = \sum_{\ell=1}^{L}\big(12n_\ell d^2 - 8rd^2 + 2n_\ell^2d\big)

multiply-adds, against $L(12Td^2 + 2T^2d)$ without merging.

∂ Proof of the merging cost

By the global case of the attention cost theorem above, the attention branch on $n_\ell$ tokens costs $4n_\ell d^2 + 2n_\ell^2d$ , and the MLP on $n_\ell - r$ tokens costs $8(n_\ell - r)d^2$ . Their sum is the term shown.

The matching itself, about $n_\ell^2d_k/4$ multiply-adds for the similarities, is $1/(8h)$ of the attention term $2n_\ell^2d$ and is left out.

Since $n_\ell$ falls with $\ell$ , the bound is tightest in the last block, where it reads $2r \le T - (L-1)r - 1$ , that is $(L+1)r \le T - 1$ . $\blacksquare$

In plain words, at the largest allowed $r$ the token count falls almost linearly from $T$ to nearly zero. The linear layers then do about half their usual work, and the attention scores, which go with $n_\ell^2$ , about a third. When $r$ is larger than the bound allows, ToMe's code merges fewer tokens in the last blocks, as many as the bound permits. The cell evaluates the theorem for the two models of ToMe's headline result.

Python · cell 9

📌 ToMe on large ViTs, off the shelf (Bolya et al. 2023, Table 9)

ViT-L/16 at 512 pixels reads $T = 1{,}025$ tokens through 24 blocks. The largest constant schedule is $r = \lfloor 1{,}024/25\rfloor = 40$ , which ends with 65 tokens. The blocks fall from 361G to 182G multiply-adds, 1.98 times fewer. The measured throughput on a V100 rose from 12.8 to 26.3 images per second, 2.05 times, while ImageNet top-1 went from 88.06% to 87.80%.

ViT-H/14 at 518 pixels reads 1,370 tokens through 32 blocks. At $r = 40$ it ends with 90 tokens, does 1.98 times fewer multiply-adds, runs 2.06 times faster (4.7 to 9.8 images per second), and goes from 88.55% to 88.25%.

Both are SWAG models (Singh et al. 2022), weakly supervised by predicting the hashtags of 3.6 billion public images, and both were used without any retraining.

The measured speedup tracks the count because a plain ViT's work is large dense matrix multiplies. Those are limited by arithmetic rather than by memory, as the section on latency below makes precise. Smaller models lose more. Off the shelf at twice the speed, ViT-B, S, and Ti drop about 4 to 5 points. Training with merging in place recovers much of that. Fine-tuned with ToMe at twice the throughput, MAE models lose 0.4 points for ViT-H, 0.6 for ViT-L, and 1.7 for ViT-B. Bolya et al. attribute the gentler loss of large models to their depth, which lets features change more gradually between merges.

🔬 Pruning instead of merging

Pruning scores the tokens and deletes the lowest. EViT (Liang et al. 2022) scores each image token by how much the class token attends to it, averaged over the heads, $\bar a_i = \frac1h\sum_{j=1}^h(\mathbf{A}_j)_{0i}$ , with $\mathbf{A}_j$ the attention matrix of head $j$ . In blocks 4, 7, and 10 of DeiT-S it keeps the top 70% of the image tokens and fuses the rest into one token, their $\bar a$ -weighted sum, so about a third of the image tokens reach the end. Trained with this in place, DeiT-S went from 4.6G to 3.0G multiply-adds and ran 50% faster, for a drop from 79.8% to 79.5% top-1. Deleting the same tokens from a trained DeiT-S without retraining costs 1.3 points instead, which is why pruning methods train.

DynamicViT (Rao et al. 2021) learns the score, with a small prediction module at three depths. It keeps training differentiable with Gumbel-softmax masks, a smooth stand-in for a hard keep-or-drop choice, that block attention to the dropped tokens. Pruning 66% of the tokens in three steps cut FLOPs by 31 to 37% and raised throughput by over 40%, for at most 0.5 points.

On DeiT-S trained from scratch, ToMe with $r = 13$ matched DynamicViT, 79.4% at 2.7G and 1,552 images per second against DynamicViT's 79.3% at 2.9G and 1,505 on the same V100, with no extra parameters. The unreduced model is 79.8% at 4.6G and 930 images per second (Bolya et al. 2023, Table 4).

method	decides by	needs training	tokens per image	survivors keep a grid position	reported result
bigger patches (FlexiViT)	position only	one model trained for many patch sizes	fixed, $(P/P')^2$ of the original	yes, on a coarser grid	one set of weights for patch sizes 8 to 48
pruning by class attention (EViT)	the class token's attention	yes	fixed keep rate	kept tokens yes, the fused token no	DeiT-S 50% faster for 0.3 points
learned pruning (DynamicViT)	a learned score	yes, fine-tuned from a trained model	fixed keep ratio	yes	31 to 37% fewer FLOPs for at most 0.5 points
merging (ToMe)	similarity of keys	no, though training helps	fixed, $T - rL$ at the end	no, a merged token spans patches	ViT-L and ViT-H 2 times faster for 0.26 and 0.30 points

Merging has a cost that classification hides. A merged token can stand for patches far apart, so the output is no longer a grid, and a dense head has nothing to put at each pixel. ToMe for Stable Diffusion (Bolya and Hoffman 2023) handles this by unmerging. It merges tokens before a layer, then copies each merged output back to every token it absorbed before the residual addition. By default it does this only around self-attention, and only in the highest-resolution blocks. That keeps the full grid and still removes up to 60% of the tokens there. Pruning keeps the survivors on the grid but leaves holes, and it needs a class token, or some other score, to rank by. And a vision-language model must give every image token a row and a column for its positions. That is why §31.5, when it compresses image tokens for a language model, compares merging by content with merging by position.

FLOPs are not latency

Every comparison so far counted multiply-adds. Wall-clock time is a different matter, for the reason §11.2 made precise. An operation's arithmetic intensity is the FLOPs it does per byte it moves to and from memory. A GPU's ridge point is its peak FLOPs per second divided by its memory bandwidth in bytes per second, about 295 FLOPs per byte for an H100 in dense bfloat16. An operation whose intensity is below the ridge point is memory-bound, so its speed is set by memory bandwidth instead of arithmetic. Here are three vision operators measured that way.

✏️ Arithmetic intensity of three vision operators

Take $T$ positions and $d$ channels in bfloat16, 2 bytes per number, and count only the activations read and written. These are true FLOPs, two per multiply-add.

A 7 by 7 depthwise convolution does $2\cdot49Td$ FLOPs and must at least read and write $Td$ numbers each, $4Td$ bytes. Its intensity is $98/4 = 24.5$ , an order of magnitude below the ridge. It is memory-bound.
A 1 by 1 convolution from $d$ to $4d$ does $8Td^2$ FLOPs on about $10Td$ bytes of activations, an intensity near $0.8d$ . That is 77 at $d = 96$ and 614 at $d = 768$ , so it becomes compute-bound as the width grows.
Window attention fused into one kernel does $4M^4d_k$ FLOPs per head per window on $8M^2d_k$ bytes of queries, keys, values, and outputs, an intensity of $M^2/2$ . That is again 24.5 for $M = 7$ .

Global attention over $T$ tokens has an intensity of about $T/2$ and is compute-bound at any useful resolution. Windows trade FLOPs for memory traffic, and the shifts, rolls, and partitions add pure data movement on top.

That is why the ConvNeXt paper notes that models with depthwise convolutions are known to run slower than their FLOPs suggest, and why it compares measured throughput too. On a V100, ConvNeXt-T ran 774.7 images per second against Swin-T's 757.9, nearly equal. On an A100 it ran 1,943.5 against 1,325.6, 47% faster (Liu et al. 2022b, Appendix E). That A100 test used TF32, a reduced-precision format that the A100's matrix units run fast, and PyTorch's channels-last memory layout, which stores the channels of each position next to each other.

The FLOPs and the networks were the same. The gap depends on the GPU, the number format, and the memory layout, because memory-bound operators are only as fast as their kernels, and standard convolutions have far better kernels than window shuffles. The same fact explains why Hiera got faster by deleting specialized modules, and why mobile hybrids optimize latency on the target chip directly. Always measure on your hardware.

How to choose a backbone

📌 Which backbone, for what

situation	first choice	why
detection or segmentation on a GPU, strong pretrained weights available	plain ViT with self-supervised weights (DINOv2 or DINOv3, MAE), windowed attention plus a few global blocks, simple feature pyramid	the best published dense results; one backbone for many tasks; can stay frozen
high-resolution dense tasks (1024 pixels and up) trained from labels, or a head that needs a hierarchical encoder	Swin, Swin V2, or Hiera	linear cost in area, native multi-scale maps, relative bias handles windows
small or medium labeled dataset, no large pretraining, or many input resolutions	ConvNeXt (V2 if you pretrain it yourself)	locality prior, fully convolutional, simple to deploy and fine-tune at any size
real-time detection on a GPU	a DETR-style detector on a DINOv2 or DINOv3 ViT (RF-DETR, DEIMv2) or a convolutional backbone (RT-DETR)	measured latency, not FLOPs, decides
phones and edge chips	convolutional or mobile hybrids (MobileViT, FastViT, EfficientViT, small ConvNeXt, pruned HGNetv2)	depthwise and reparameterized convolutions, few memory-bound reshapes
classification or retrieval embeddings at 224 to 384 pixels	plain ViT (often a CLIP or SigLIP image tower)	global attention is cheap at a few hundred tokens and scales best
a trained plain ViT that is too slow for classification or retrieval	token merging (ToMe), off the shelf or fine-tuned with it	about 2 times the throughput of large models for a few tenths of a point, with no retraining

Five questions narrow the choice.

How many tokens will the input make? That decides whether global attention is affordable.
Are strong pretrained weights available? They usually matter more than the architecture.
Does the head need several scales natively?
What is the latency budget on the target hardware, measured rather than counted?
Will the resolution or aspect ratio change between training and deployment? Convolutions and relative biases transfer, while absolute position tables need interpolation.

⚠️ Where hierarchical backbones go wrong in practice

Forgetting to roll the output back after a shifted layer writes every token's output onto another token. Nothing crashes, so the bug can survive until someone compares accuracy.
Rolling in the wrong direction, or labeling regions from the unshifted map, masks the wrong pairs (Exercise 27.5b catches both).
Swin's code sets the window to the map size and the shift to zero whenever the map is no larger than the window. Stage 4 at 224 pixels has a 7 by 7 map and $M = 7$ , so its "shifted" blocks are global attention. The code also pads the map at the bottom right when a side is not a multiple of $M$ , which matters for detection inputs such as 800 by 1333.
The mask has one copy per window, so it must broadcast over heads and repeat over the batch in the right order. Mixing up masks across heads or images is easy.
In ConvNeXt, LayerNorm normalizes over channels. In PyTorch's channels-first layout that means a custom norm or a permute, and normalizing over the spatial axes instead is a common silent bug.
Comparing backbones by FLOPs alone misleads, since depthwise convolutions and window shuffles are memory-bound. ConvNeXt-T's throughput lead over Swin-T was 2% on a V100 and 47% on an A100.

🧭 What to remember

Dense prediction wants feature maps at strides 4 to 32. Halving the map and doubling the channels at each stage keeps every block's cost constant, since $H_sW_sd_s^2$ does not change.
Window attention costs $4Td^2 + 2M^2Td$ against $4Td^2 + 2T^2d$ , linear in image area. At Swin-T's first stage that is 0.145G against 2.00G multiply-adds.
Alternating regular and shifted windows grows the receptive field by $M$ per layer along each axis. Rolling by $-\lfloor M/2\rfloor$ , masking pairs from different regions, and rolling back computes shifted-window attention exactly with the same window count. Swin's relative bias is a $(2M-1)^2$ table per head indexed by the query-minus-key offset.
Swin-T has 28,288,354 parameters and 4.49G multiply-adds, only about 3% of them in attention scores. ConvNeXt matched it with a modernized ResNet (76.1% to 82.0%), which showed that the training recipe and local computation, not attention, explained most of the gain.
Hybrids put convolutions early and attention late (CoAtNet, mobile designs). With strong self-supervised pretraining a plain ViT with windows and a few global blocks is again the best dense backbone (ViTDet, Hiera, DINOv3), the same design that runs the encoders of SAM and Qwen2.5-VL. FLOPs do not predict latency, since depthwise convolutions and window attention are memory-bound.
A trained plain ViT can also run on fewer tokens. ToMe merges the $r$ most similar pairs of a bipartite split by key similarity in every block, averages by size, and adds $\ln s$ to the scores, which is exact for duplicated tokens. At the largest constant $r$ it halves the work and doubled the throughput of ViT-L and ViT-H for 0.26 and 0.30 points, with no retraining, while pruning (EViT, DynamicViT) must train.
Next, §27.6 puts heads on these feature maps and turns them into boxes and masks, with DETR's set loss, COCO's metrics, and SAM.

Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Equations 1 and 2 for the cost, Figure 4 for the cyclic shift, and Tables 4 to 6 for the ablations of shifting, position bias, and implementations.
Liu et al. (2022a). Swin Transformer V2: Scaling Up Capacity and Resolution. Res-post-norm, scaled cosine attention, and the log-spaced continuous position bias that let windows change size.
Liu et al. (2022b). A ConvNet for the 2020s. The step-by-step modernization of ResNet-50 (Figure 2 and Table 10) is a model of a controlled architecture study.
Woo et al. (2023). ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. Masked pretraining for ConvNets and global response normalization.
Dai et al. (2021). CoAtNet: Marrying Convolution and Attention for All Data Sizes. Depthwise convolution and attention unified by a relative bias, and the C-C-T-T layout.
Li et al. (2022a). Exploring Plain Vision Transformer Backbones for Object Detection. ViTDet, the simple feature pyramid, and window attention with four global blocks.
Ryali et al. (2023). Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles. How MAE pretraining lets a hierarchical ViT drop its vision-specific modules and get faster.
Bolya et al. (2023). Token Merging: Your ViT But Faster. Bipartite soft matching and proportional attention, with the ablations (Table 1) behind every design choice and the full speed and accuracy tables in Appendix A.

Made for larger screens

Hierarchical and efficient vision backbones

Why dense prediction wants a pyramid

Window attention, and what it costs

Window partition and its inverse

Shifted windows let information cross borders

The cyclic shift, and the mask that makes it exact

The shifted-window attention mask

Relative position bias in a window

Swin's relative position index

Patch merging, and Swin counted to the last parameter

The cost of a Swin backbone

A Swin block in PyTorch

Shifted window attention, end to end

ConvNeXt rebuilds a ResNet with Transformer-era choices

A ConvNeXt block in NumPy

Hybrids put convolutions early and attention late

Plain ViTs return with self-supervised pretraining

Fewer tokens instead of cheaper attention

One step of token merging

FLOPs are not latency

How to choose a backbone

Read more

Watch next