Chapter 28 · Section 28.6 · 80 min

Evaluating representations

This section is about measuring what a pretrained image encoder has learned. The encoders of this chapter were trained without labels, so their own training loss says little about whether they will be useful. To find out, you attach a small learner that does see labels, train it on a downstream task (the task you actually care about), and score it. The rules that fix how much that learner may do are called an evaluation protocol. Changing the protocol can reverse which encoder looks best.

Here is such a reversal in real numbers. The benchmark is ImageNet, 1.28 million training photographs in 1,000 classes. It is scored on its 50,000 validation images by top-1 accuracy, the fraction of images whose highest-scoring class is the right one. Take two encoders trained without labels, both Vision Transformers (ViTs). A masked autoencoder (MAE) learns by repainting the hidden patches of an image (§28.5). DINOv2 learns by making a student network match a slowly updated copy of itself on different crops of the same image (§28.4).

Freeze both, so their weights never change, and read ImageNet off their features in two ways. A k-nearest-neighbor vote labels each test image by the labels of the training images closest to it in feature space. A linear probe trains one linear classifier on the features. The largest MAE model (ViT-H/14, the huge size, with 14 by 14 pixel patches) gets 49.4 percent by the vote and 76.6 by the probe. DINOv2's giant model (ViT-g/14) gets 83.5 and 86.5 (Oquab et al. 2023, Table 4). DINOv2 looks far better. Now let every weight of the MAE change while it trains on ImageNet's labels, which is called fine-tuning. It reaches 86.9 (He et al. 2022), the same league.

The reversal also happens across tasks. As of late 2026, the self-supervised DINOv3 scores 88.4 with a linear probe on ImageNet. Two encoders trained to match images with their captions, the approach of CLIP, score a little higher. They are SigLIP 2 at 89.1, which scores each image-caption pair with its own sigmoid (§30.4), and Perception Encoder at 89.3. Now ask the same frozen features to label every pixel of a scene. On the semantic segmentation benchmark ADE20k, scored by mIoU (the average overlap between predicted and true regions), DINOv3 leads by 13 to 17 points, 55.9 against 42.7 and 38.9 (Siméoni et al. 2025, Tables 3 and 7).

So which encoder is better? There is no answer until you say what the encoder is for, because each protocol asks a different question. §28.1 defined three protocols, the linear probe, k-NN, and fine-tuning, just well enough to read a results table. This section takes them apart. For each one it says what it computes, what property of the features it measures, how to run it correctly, and which details move the number. Then it explains why protocols disagree, puts honest error bars on top-1 accuracy, judges a representation with no labels at all, and chooses an encoder for a real task. A team that misreads a results table ships the wrong encoder, which is why interviewers for vision and multimodal roles ask about all of this.

Some notation for the whole section. The pretrained encoder is $f_\theta$ , a backbone network followed by a pooling step that turns its outputs into one vector. For an image $x$ , the vector $\mathbf{h} = f_\theta(x) \in \R^{d}$ is its representation, the vector every protocol reads. For a ResNet-50 $d = 2048$ , and for a ViT-B $d = 768$ . Most methods of this chapter train with an extra small network on top, the projection head $g_\theta$ , and the training loss sees its output $\mathbf{z} = g_\theta(\mathbf{h})$ . The head is thrown away before evaluation, because §28.2 measured that $\mathbf{h}$ transfers better than $\mathbf{z}$ . So evaluation reads $\mathbf{h}$ . A downstream task has $K$ classes and a labeled set of $n$ images. Frozen means the encoder's weights are not updated during evaluation.

Every protocol is a question with a budget

A protocol fixes how much work a downstream learner may do with the frozen features. That budget decides which property of the features the score measures.

k-NN trains nothing. It asks whether an image's nearest neighbors in feature space share its label. That is a question about local geometry.
A linear probe trains one matrix. It asks whether flat cuts through feature space (hyperplanes) separate the classes. That is a question about global linear structure.
An attentive probe trains one attention layer over the patch tokens, the per-patch output vectors of a ViT. It asks whether the information is somewhere in the tokens, even if no single pooled vector holds it.
Partial and full fine-tuning train some or all of the encoder. They ask how good a starting point the weights are, which is a question about optimization as much as about features.
Dense probes train a linear map on every patch token. They ask whether each patch carries local meaning or geometry.

A bigger budget can extract more. It also measures less about the representation and more about the learner. Two toy geometries make the first two protocols concrete.

✏️ Two toy representations, by hand

Suppose a representation maps a two-class dataset to points in the plane.

Disk and ring. Class A fills the unit disk. Class B fills the ring between radii 1.5 and 2.5. A gap of 0.5 separates them, so every point's nearest neighbors are of its own class. Even a 1-nearest-neighbor rule is perfect. Yet no line separates a disk from a ring around it. The best a line can do is cut off the part of the ring beyond a tangent to the disk. With the two classes equally common, that gives about two thirds accuracy.

Long blobs. Along the first axis the class means are 1.5 apart and each class has standard deviation 1. Along the second axis both classes have standard deviation 4, and that axis says nothing about the label. The best classifier thresholds the first coordinate halfway between the means. The threshold is 0.75 standard deviations from each mean, so the accuracy is $\Phi(0.75) = 0.773$ , where $\Phi$ is the standard normal distribution function. That classifier is linear. A 1-nearest-neighbor rule measures distance mostly along the useless second axis, whose spread is four times larger. Its nearest neighbor is chosen mostly by the second coordinate, so its label is a weak guess.

The first geometry is one that k-NN reads and a linear probe cannot. The second is the reverse.

The cell measures both, with 400 labeled points per geometry and 2,000 test points. The toy features are not on a sphere, so it uses the Euclidean distance. Real embeddings are first scaled to unit length, as the k-NN section explains.

Python · cell 1

On the disk and ring, k-NN is perfect for $k = 1$ and $k = 5$ . It gets 0.998 at $k = 51$ , where a few neighbors reach across the gap. The logistic-regression probe gets 0.393, worse than always guessing one class. The reason is that logistic regression does not minimize the error rate. It minimizes the cross-entropy, a smooth stand-in for it. When no line separates the classes, the line with the lowest cross-entropy can be a poor classifier, here far below the best line's two thirds.

On the long blobs the linear probe gets 0.760, close to the optimum 0.773, while 1-NN gets 0.671. More neighbors help k-NN (0.717 at $k = 5$ , 0.753 at $k = 51$ ), because the noise along the useless axis averages out over many neighbors. This is the first lesson of the section. Directions with high variance and no label information hurt k-NN, and a linear probe learns to ignore them. Classes that are locally coherent but not linearly arranged hurt a linear probe, and k-NN does not care.

The widget runs both protocols on four toy geometries. Each dot is one labeled image's feature vector, and the shading shows each protocol's prediction everywhere in the plane.

The "collapsed" geometry is the failure §28.3 warned about. Two classes the encoder should have separated land on top of each other. No downstream learner of any budget can recover what the representation threw away. The widget also reports the effective rank, which counts how many directions carry the spread of the features. §28.3 defined it from the singular values, and it is restated later in this section. Here it falls to 1.1, against 2.0 for the clustered features. We come back to that number as a check that needs no labels.

The linear probe

The linear probe is the headline number of self-supervised vision. §28.1 traced its history, from 40 to 50 percent on ImageNet for early pretext tasks to 88.4 for DINOv3. It looks trivial to run. It is also easy to run in a way that silently changes the number, so here is the precise recipe.

📐 Linear probe

Freeze the encoder $f_\theta$ in evaluation mode, so that every BatchNorm layer uses its stored statistics and dropout is off, and stop gradients into it. For each labeled image extract $\mathbf{h} \in \R^{d}$ . Say which vector $\mathbf{h}$ is. It can be the class token, the average of the patch tokens, their concatenation, or a concatenation over several of the last blocks. Optionally standardize each coordinate with a BatchNorm that has no learnable scale or shift. Train only $\mathbf{W}_{\text{probe}} \in \R^{d\times K}$ and $\mathbf{b}_{\text{probe}} \in \R^{K}$ , with logits $\mathbf{W}_{\text{probe}}^\top\mathbf{h} + \mathbf{b}_{\text{probe}}$ , by cross-entropy on the training set. Training usually uses random-resized-crop and flip augmentation. Evaluate on a center crop (on ImageNet, resize the short side to 256 pixels and crop 224). Choose the learning rate, epochs, and weight decay on a validation split carved from the training set, never on the test set. Report top-1 accuracy on the test set.

Training the probe is a convex problem, because the features are fixed and the classifier is linear. For a ViT-L ( $d = 1024$ ) on ImageNet the probe has about a million parameters ( $d \times K$ ), against 1.28 million training images, so it needs little regularization. Regularizers that perturb the inputs only blur the frozen geometry the probe is meant to read. An example is Mixup, which blends two images and their labels. Published recipes turn these off.

Frozen means evaluation mode

BatchNorm, introduced in §0.7, standardizes each feature by subtracting a mean and dividing by a standard deviation. In training mode it uses the current batch's statistics and updates stored running averages. In evaluation mode it uses the stored averages, which are fixed numbers. A non-affine BatchNorm has no learnable scale and shift after the standardization.

MAE puts a non-affine BatchNorm between the frozen features and the probe, following Doersch et al. (2015). Does that make the probe nonlinear, or more powerful? No.

∴ BatchNorm before a probe changes nothing but conditioning

Let $\boldsymbol\mu \in \R^{d}$ and $\mathbf{s} \in \R^{d}$ , with every $s_j > 0$ , be the frozen mean and standard deviation that a non-affine BatchNorm uses at test time. For every probe $(\mathbf{W}, \mathbf{b})$ on the normalized features there is a probe on the raw features with identical logits,

\mathbf{W}^\top\operatorname{diag}(\mathbf{s})^{-1}(\mathbf{h} - \boldsymbol\mu) + \mathbf{b} = \mathbf{W}'^\top\mathbf{h} + \mathbf{b}', \qquad \mathbf{W}' = \operatorname{diag}(\mathbf{s})^{-1}\mathbf{W}, \quad \mathbf{b}' = \mathbf{b} - \mathbf{W}'^\top\boldsymbol\mu ,

and conversely. The two parameterizations define the same set of classifiers. Only the optimization differs.

Proof. Expand the left side to $\mathbf{W}^\top\operatorname{diag}(\mathbf{s})^{-1}\mathbf{h} - \mathbf{W}^\top\operatorname{diag}(\mathbf{s})^{-1}\boldsymbol\mu + \mathbf{b}$ . A diagonal matrix is symmetric, so $\mathbf{W}^\top\operatorname{diag}(\mathbf{s})^{-1} = \big(\operatorname{diag}(\mathbf{s})^{-1}\mathbf{W}\big)^\top = \mathbf{W}'^\top$ . The expression is therefore $\mathbf{W}'^\top\mathbf{h} + (\mathbf{b} - \mathbf{W}'^\top\boldsymbol\mu)$ . For the converse take $\mathbf{W} = \operatorname{diag}(\mathbf{s})\mathbf{W}'$ and $\mathbf{b} = \mathbf{b}' + \mathbf{W}'^\top\boldsymbol\mu$ , which works because every $s_j$ is positive. ∎

In plain words, standardizing the features first is a change of coordinates for the probe's weights. It cannot reach any classifier that was not reachable before. What it changes is how gradient descent sees the problem. Coordinates of pretrained features can differ in scale by large factors. One learning rate is then too large for some coordinates and too small for others. Standardizing lets one learning rate work for every checkpoint. MAE adopted it for exactly that reason, so its many ablations could share one probe recipe without a learning-rate search (He et al. 2022, Appendix A). The identity holds only in evaluation mode, where $\boldsymbol\mu$ and $\mathbf{s}$ are fixed. An interviewer who asks "does the BatchNorm make it a nonlinear probe?" wants this fold.

✏️ Folding a BatchNorm into a probe, by hand

Take $d = K = 2$ and frozen statistics $\boldsymbol\mu = (1, -2)$ and $\mathbf{s} = (2, 0.5)$ . The probe on the standardized features has columns $\mathbf{w}_1 = (1, 2)$ and $\mathbf{w}_2 = (-1, 0)$ and bias $\mathbf{b} = (0.5, 0)$ .

For $\mathbf{h} = (3, -1)$ the standardized vector is $\big((3-1)/2,\ (-1+2)/0.5\big) = (1, 2)$ . The logits are $1 + 4 + 0.5 = 5.5$ and $-1 + 0 + 0 = -1$ .

Folding divides row $j$ of $\mathbf{W}$ by $s_j$ , so $\mathbf{w}'_1 = (0.5, 4)$ and $\mathbf{w}'_2 = (-0.5, 0)$ . The new bias is $\mathbf{b}' = \mathbf{b} - \mathbf{W}'^\top\boldsymbol\mu = \big(0.5 - (0.5 - 8),\ 0 - (-0.5)\big) = (8, 0.5)$ . On the raw vector the logits are $1.5 - 4 + 8 = 5.5$ and $-1.5 + 0 + 0.5 = -1$ , the same.

The identity also shows what goes wrong when "frozen" is taken to mean only "no gradients".

The cell builds a one-layer encoder whose BatchNorm holds statistics from pretraining. It then runs 100 probe-training forward passes over downstream data with a different mean and scale, with the encoder left in training mode.

Python · cell 2

No weight of the encoder received a gradient. Yet a running mean moved by 2.04, and the evaluation-mode features of the same five images changed by up to 4.79. The probe was trained on a different encoder from the one that was reported.

Which features, and which hyperparameters

"The linear probe accuracy" is really a family of numbers. The choices inside the definition move it by points.

📌 Choices inside the probe move the number

Which vector. DINO's ViT-S/16 gets 76.1 percent from the last block's class token. It gets 77.0 from the class tokens of the last four blocks concatenated. For ViT-B/16, the class token concatenated with the average patch token gives 78.2, against 78.0 for the class token alone (Caron et al. 2021, Appendix F).
Which layer. AIM, a 7-billion-parameter model trained to predict image patches in order, gets 82.4 from its last layer and 84.0 from its best one, measured with an attentive probe (El-Nouby et al. 2024, Table 7). Perception Encoder is trained only to match images and video frames with their captions. The intermediate layers of its largest model, PE-core G, rival AIMv2-3B, a strong encoder trained by captioning, on language tasks, and DINOv2-g on spatial ones (Bolya et al. 2025).
Which grid. DINOv2's probe searches 13 learning rates, 1 or 4 output layers, and with or without pooled patch tokens. That is 52 configurations, all trained at once on features from a single forward pass. It reports the highest accuracy on the validation set, "as is common practice" (Oquab et al. 2023, Appendix B.3).

That last phrase deserves a pause. On ImageNet, "the validation set" is the test set everyone reports on.

⚠️ Tuning on the test set

If you choose probe settings by their accuracy on the images you then report on, you report the luckiest configuration, and the number is too high. Suppose the 52 configurations were equally good and made independent errors of standard deviation $\sigma$ around their common true accuracy. The best one would then be reported about $2.26\sigma$ too high on average, because 2.26 is the expected maximum of 52 independent standard normals. With $\sigma = 0.18$ points, the test-set noise computed later in this section, that is about 0.4 points.

The real bias is smaller, because the configurations share the same 50,000 images and their errors are strongly correlated. It is never zero, and on small datasets it is large. The exercise below shows 5 points on a 600-image test set. The honest recipe holds out part of the training set for selection, as BYOL and the VTAB benchmark do. VTAB then refits on all its training labels with the chosen settings, as the exercise below does. Papers that do select on ImageNet validation also report two cleaner test sets. ImageNet-ReaL relabels the validation images, and ImageNet-V2 is a fresh test set collected the same way. DINOv2 reports both for exactly this reason, since most self-supervised methods were developed against ImageNet validation accuracy, and DINOv3 reports them too.

Core

A linear probe done right

Implement linear_probe_done_right(train_h, train_y, test_h, test_y, Cs, seed=0). It tunes a logistic-regression probe honestly and returns (best_C, test_accuracy).

Carve a validation split from the training set. Compute perm = np.random.default_rng(seed).permutation(n), take perm[:n // 5] as validation and the rest as the fitting part.
Standardize every split with the mean and standard deviation (ddof=0) of the fitting part.
For each C in Cs, fit LogisticRegression(C=C, max_iter=2000) on the fitting part and score it on the validation part. C is scikit-learn's inverse regularization strength, so a small C means strong weight decay. Keep the first C with the best validation accuracy.
Refit with that C on the whole training set, now standardized with the whole training set's statistics, and score it on the test set.

The setup's features have coordinates on very different scales, and its test images come from a slightly different camera, so each wrong choice gives a visibly different number.

import numpy as np
from sklearn.linear_model import LogisticRegression

def linear_probe_done_right(train_h, train_y, test_h, test_y, Cs, seed=0):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

On this data the honest procedure picks $C = 0.03$ and scores 77.5 percent. Choosing $C$ by test accuracy would pick $C = 0.001$ and report 82.3, about five points of pure optimism from looking at the answers.

What a linear probe can and cannot read

A linear probe can only combine the directions the features actually use. Suppose every representation lies on one line through feature space, so the matrix of representations has rank 1. Then every class score the probe computes is a fixed multiple of one number, plus a bias. Take four balanced classes of 25 images each, and fit the one-hot targets by least squares. They cannot all be fit, and the theorem below says the squared error is at least 50, or 0.5 per image. Here is the general statement. It also motivates the label-free check near the end of the section.

∴ A linear probe is limited by the rank of the representation

Stack the representations of $n$ labeled images as rows of $\mathbf{H} \in \R^{n\times d}$ with $\operatorname{rank}(\mathbf{H}) = r$ . Stack their one-hot labels as rows of $\mathbf{Y} \in \{0, 1\}^{n\times K}$ , with $n_c$ images in class $c$ . For every probe, the prediction matrix $\mathbf{P} = \mathbf{H}\mathbf{W}_{\text{probe}} + \mathbf{1}\mathbf{b}_{\text{probe}}^\top$ satisfies

\lVert\mathbf{Y} - \mathbf{P}\rVert_F^2 \;\ge\; \sum_{j > r+1}\sigma_j(\mathbf{Y})^2 \;=\; \text{the sum of the } K - r - 1 \text{ smallest class counts } n_c ,

where $\sigma_j$ are the singular values in descending order, and the sum is empty when $r + 1 \ge K$ .

∂ Proof of the rank bound

The columns of $\mathbf{H}\mathbf{W}_{\text{probe}}$ are combinations of the columns of $\mathbf{H}$ , so $\operatorname{rank}(\mathbf{H}\mathbf{W}_{\text{probe}}) \le r$ . Adding the rank-one matrix $\mathbf{1}\mathbf{b}_{\text{probe}}^\top$ raises the rank by at most one, so $\operatorname{rank}(\mathbf{P}) \le r + 1$ .

The Eckart-Young theorem of §0.10 says that the closest matrix of rank at most $r + 1$ to $\mathbf{Y}$ in Frobenius norm leaves exactly the energy of the discarded singular values, $\sum_{j > r+1}\sigma_j(\mathbf{Y})^2$ . No matrix of that rank does better.

To find those singular values, note that each row of $\mathbf{Y}$ has a single 1. So $Y_{ic}Y_{ic'} = 0$ whenever $c \ne c'$ , and $(\mathbf{Y}^\top\mathbf{Y})_{cc'} = \sum_i Y_{ic}Y_{ic'} = n_c\,\mathbb{1}[c = c']$ . Thus $\mathbf{Y}^\top\mathbf{Y} = \operatorname{diag}(n_1, \dots, n_K)$ , whose eigenvalues are the class counts. The squared singular values of $\mathbf{Y}$ are the eigenvalues of $\mathbf{Y}^\top\mathbf{Y}$ . Sorted in descending order, the ones beyond position $r + 1$ are the $K - r - 1$ smallest counts. ∎

In plain words, take balanced classes of $n/K$ images each. A least-squares probe on a rank- $r$ representation then leaves a squared error of at least $(K - r - 1)/K$ per image. A representation that has collapsed to 3 dimensions cannot fit ten one-hot targets, whatever the probe. Check it on the opening example. With $K = 4$ and $r = 1$ the floor is $(4 - 1 - 1)/4 = 0.5$ per image, the 50 over 100 images promised above.

The bound is about the least-squares fit, and two caveats keep it honest. First, a classifier only needs the right argmax, the class with the highest score, which is a weaker requirement. Ten tight clusters spaced around a circle have rank 2. Yet the ten affine scores $\mathbf{h}^\top\boldsymbol\mu_c - \tfrac12\lVert\boldsymbol\mu_c\rVert^2$ , one per cluster center $\boldsymbol\mu_c$ , pick the nearest center and classify them perfectly. Low rank crowds the classes together, so they are easy to confuse once there is noise, but it is not by itself a ceiling on accuracy. Second, high rank is necessary for a good least-squares fit but not sufficient. Random features have full rank and know nothing about the labels. The authors who turned rank into a label-free score say the same, that rank is necessary but not sufficient (Garrido et al. 2023).

The cell checks the bound on random rank- $r$ projections of ten noisy clusters, and then classifies the ten clusters on a circle.

Python · cell 3

The least-squares error never goes below its floor (1614.8 against 1548 at $r = 1$ , 1036.6 against 942 at $r = 4$ ). The argmax accuracy of the random projections climbs from 0.200 at $r = 1$ to 0.995 at $r = 8$ , while the circle reaches 1.0 at rank 2. Rank bounds what a linear probe can fit. What it can classify also depends on how the classes are arranged. In practice, a representation whose rank has fallen far below the number of classes is a red flag, and a high rank alone proves nothing.

The weighted k-NN classifier

k-NN is the protocol that needs no training at all. That is why it is the number people watch during pretraining.

📐 Weighted k-NN evaluation

Freeze $f_\theta$ in evaluation mode. Compute the unit-length features $\hat{\mathbf{h}}_i = \mathbf{h}_i/\lVert\mathbf{h}_i\rVert$ of all $n$ labeled training images and store them with their labels $y_i$ . For a test image with unit feature $\hat{\mathbf{h}}$ , compute the cosine similarities $s_i = \hat{\mathbf{h}}_i^\top\hat{\mathbf{h}}$ and let $\mathcal{N}_k$ be the indices of the $k$ largest. Predict

\hat y = \arg\max_{c}\ \sum_{i \in \mathcal{N}_k} e^{s_i/\tau}\,\mathbb{1}[y_i = c],

with a temperature $\tau > 0$ . Here $k$ counts neighbors. Instance discrimination, which trains every image to be its own class (§28.1), used $k = 200$ and $\tau = 0.07$ (Wu et al. 2018). DINO uses $k = 20$ , the best value in its runs, and the same $\tau$ (Caron et al. 2021).

In plain words, each of the $k$ most similar training images votes for its own label, and closer images get exponentially larger votes. For example, with $\tau = 0.07$ a neighbor at cosine 0.9 outvotes one at cosine 0.8 by a factor of $e^{0.1/0.07} \approx 4.2$ .

Two limits show what $\tau$ does. Divide every weight by the largest one, $e^{s_{\max}/\tau}$ , which does not change the argmax. Each neighbor's weight becomes $e^{-(s_{\max} - s_i)/\tau}$ . As $\tau \to 0$ , every neighbor except the most similar gets a weight that goes to 0, so the rule becomes 1-NN (barring exact ties). As $\tau \to \infty$ , every weight tends to 1, so the rule becomes an unweighted vote of $k$ neighbors. In between, it is a kernel classifier, a vote in which each training point's weight is a fixed function of its similarity to the query. The kernel $e^{s/\tau}$ is no accident. InfoNCE, the contrastive loss of §7.2, is a softmax over exactly these exponentials of cosine similarities. A model trained with it is trained to make these similarities informative.

The costs are low. The protocol needs no augmentation, no learning rate, and one forward pass over the training set. It does need memory for the feature bank, the stored training features. ImageNet's 1.28 million training images at $d = 768$ in float16 take $1.28\times10^6 \times 768 \times 2$ bytes, about 2.0 GB. The search is the exact top- $k$ search of §7.4. It runs over blocks of the bank with a running merge, so the full similarity matrix never exists.

Its weaknesses are the ones the long blobs showed. A few high-variance directions that carry no label can dominate the cosine. The result also depends on $k$ , on $\tau$ , and on whether the features were normalized. Across real encoders, the gap between k-NN and the linear probe follows a telling pattern.

Warm-up

Weighted k-NN classifier

Implement knn_classify(train_h, train_y, test_h, k, tau, n_classes) exactly as the definition says, and return the predicted labels as an integer array of shape (n_test,).

Normalize both banks to unit length and compute all cosine similarities with one matrix product.
Keep the $k$ largest similarities in each test row (use np.argpartition). If k exceeds the number of stored features, use all of them.
Weight each neighbor's vote by $e^{s/\tau}$ , sum the votes per class, and predict the class with the largest total. Break ties toward the smaller label, which argmax does.

The votes must not overflow for a small temperature such as $\tau = 0.001$ , where $e^{1/\tau}$ is far beyond the largest float.

The data here is small, so you do not need the blocked search of §7.4.

import numpy as np

def knn_classify(train_h, train_y, test_h, k, tau, n_classes):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Attentive probes

Why is MAE's linear probe so much worse than its fine-tuning? Part of the answer is the pooling. A linear probe reads one vector per image, either the class token or the average of the patch tokens. MAE's loss is computed on the reconstructed pixels of masked patches, so nothing ever asked either vector to summarize the image. An average over all patches also mixes the object with the background in proportion to their areas. An ImageNet label concerns only the patches that show the object. The authors of the context autoencoder, a masked-modeling method, made this argument and proposed letting the probe choose which patches to read (Chen et al. 2022).

📐 Attentive probe

Let $\mathbf{H} \in \R^{T\times d}$ be the frozen patch tokens of one image, with the class token and any registers (extra learned tokens, §27.4) removed. An attentive probe with $h$ heads of width $d_k = d/h$ has a learned query $\mathbf{q} \in \R^{d}$ , key, value, and output matrices $\mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O \in \R^{d\times d}$ , and a classifier $\mathbf{W}_{\text{cls}} \in \R^{d\times K}$ . Split $\mathbf{q}$ into $h$ pieces $\mathbf{q}^{(j)} \in \R^{d_k}$ . Split the columns of $\mathbf{K} = \mathbf{H}\mathbf{W}_K$ and $\mathbf{V} = \mathbf{H}\mathbf{W}_V$ into $h$ blocks $\mathbf{K}^{(j)}, \mathbf{V}^{(j)} \in \R^{T\times d_k}$ , so head $j$ , for $j = 1, \dots, h$ , owns columns $(j-1)d_k + 1$ to $j d_k$ . Then

\mathbf{a}^{(j)} = \operatorname{softmax}\!\Big(\frac{\mathbf{K}^{(j)}\mathbf{q}^{(j)}}{\sqrt{d_k}}\Big) \in \R^{T}, \qquad \mathbf{o} = \big[\mathbf{V}^{(1)\top}\mathbf{a}^{(1)};\ \dots;\ \mathbf{V}^{(h)\top}\mathbf{a}^{(h)}\big] \in \R^{d}, \qquad \text{logits} = \mathbf{W}_{\text{cls}}^\top\mathbf{W}_O^\top\mathbf{o} \in \R^{K},

where the softmax runs over the $T$ tokens. Everything except the encoder is trained with cross-entropy, as for a linear probe.

In plain words, each head scores every patch against a learned question and takes a weighted average of the patches' values with those scores. The classifier reads the concatenated answers. This is one layer of the multi-head attention of §4.5, with a single query that belongs to the probe instead of the image. There is no query projection, because a learned vector multiplied by a learned matrix is just another learned vector.

The probe is bigger than a linear one. For a ViT-L ( $d = 1024$ ) on ImageNet it has $3d^2 + dK = 4.2$ million parameters, against the linear probe's $dK + K = 1.0$ million. It is also no longer linear in $\mathbf{H}$ , so it can read information a pooled vector hides. AIM uses it as its main protocol, and self-supervised video models use the same idea with a full cross-attention block.

The evidence is the pattern of gains.

The masked-modeling features were never that bad. The linear probe was reading the wrong summary of them. In practice, if you evaluate a masked-modeling encoder frozen, use an attentive probe, and say so when you compare it with a linear-probe number. The exercise is multi-head attention with one twist in the shapes, a single query per head, which makes it a good interview question.

Interview

An attentive probe forward pass

Implement attentive_probe(H, q, Wk, Wv, Wo, W_cls, n_heads, n_prefix=0) with no loops. The inputs are a batch of token arrays H of shape (B, T, d), a learned query q of shape (d,), the matrices Wk, Wv, Wo of shape (d, d), and W_cls of shape (d, K).

Drop the first n_prefix tokens of every image (the class token and registers).
Compute the definition above. Counting heads from 0 as in code, head j uses columns j*dk to (j+1)*dk - 1 of the keys, values, and query, the same head split as the head-splitting exercise of §0.1.
Return (logits, A), with logits of shape (B, K) and the attention weights A of shape (B, n_heads, T - n_prefix).

The softmax must be numerically stable.

import numpy as np

def attentive_probe(H, q, Wk, Wv, Wo, W_cls, n_heads, n_prefix=0):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Fine-tuning, partial fine-tuning, and LP-FT

The largest budget lets the downstream learner change the encoder itself.

📐 Fine-tuning protocols

Full fine-tuning. Initialize from $f_\theta$ , attach a task head, and train everything on the labeled data. Earlier blocks usually get smaller learning rates. With layer-wise learning-rate decay $\zeta \in (0, 1]$ and base rate $\eta$ , layer $i$ trains at $\eta\,\zeta^{L+1-i}$ . Here the embeddings are layer 0, block $\ell$ is layer $\ell$ , and the head is layer $L + 1$ (§27.4).

Partial fine-tuning. Freeze all but the last $j$ blocks and train those with the head. So $j = 0$ is a linear probe and $j = L$ is full fine-tuning.

LP-FT. Train a linear probe first. Then fine-tune everything, starting from the probed head instead of a random one.

Low-shot fine-tuning or probing. Use only a class-balanced fraction of the labels, most often 1 or 10 percent of ImageNet, about 12.8 and 128 images per class. These subsets come from a study of semi-supervised learning (Zhai, Oliver, Kolesnikov, and Beyer 2019), and SimCLR, the contrastive method of §28.2, adopted them.

A tiny example of the decay rule. A ViT-L has $L = 24$ blocks. With $\zeta = 0.75$ , the last block trains at $0.75\,\eta$ , block 12 at $0.75^{13}\eta \approx 0.024\,\eta$ , and the patch embedding at $0.75^{25}\eta \approx 0.00075\,\eta$ . The early layers, which hold the most general features, barely move.

Partial fine-tuning is the experiment that made the field stop treating the linear probe as the measure of a representation.

MAE's features are strong but nonlinear. A single trainable block reads what a hyperplane cannot, much as k-NN read the disk and ring that defeated the linear probe in the widget. Fine-tuned in full at 224 pixels, the MAE ViT-H reaches 86.9.

Comparisons of fine-tuned numbers have their own trap. A fine-tuning recipe is a long list of choices. MAE's recipe (He et al. 2022, Table 9) uses a layer-wise decay of 0.75 and Mixup 0.8. It uses CutMix 1.0, which pastes a patch of one image into another, and label smoothing 0.1, which moves a little of each target's probability onto the other classes. It also uses drop path, which randomly skips residual blocks, and RandAugment, a random sequence of image operations. Two pretrained models fine-tuned with different recipes differ in two things at once. A fair comparison fine-tunes both with the same recipe, and a careful one tunes the recipe for each.

Why fine-tuning can hurt out of distribution

Fine-tuning usually wins on test data like its training data, called in distribution (ID). It can lose badly out of distribution (OOD), on test data that has shifted, such as sketches of the same objects. The reason is exact in a small model studied by Kumar et al. (2022).

The model is a two-layer linear network $f(x) = \mathbf{v}^\top\mathbf{B}x$ . The matrix $\mathbf{B} \in \R^{d'\times d}$ is the feature extractor. It computes $d'$ features from an input in $\R^{d}$ , and $\mathbf{v} \in \R^{d'}$ is the head. Gradient descent minimizes the squared loss $\mathcal{L}(\mathbf{v}, \mathbf{B}) = \tfrac1n\lVert\mathbf{X}\mathbf{B}^\top\mathbf{v} - \mathbf{y}\rVert^2$ over $n$ ID inputs, the rows of $\mathbf{X} \in \R^{n\times d}$ .

The reason fits in one line. The gradient with respect to $\mathbf{B}$ is a combination of the training inputs, so it has no component in any direction those inputs never visit. The full proof follows.

∂ Proof that the features move only inside the training span

Write the residuals as $\mathbf{r} = \mathbf{X}\mathbf{B}^\top\mathbf{v} - \mathbf{y} \in \R^{n}$ , so the loss is $\tfrac1n\mathbf{r}^\top\mathbf{r}$ . The entry $B_{ab}$ enters residual $i$ as $v_aB_{ab}X_{ib}$ , so $\partial\mathcal{L}/\partial B_{ab} = \tfrac2n\sum_i r_i v_a X_{ib}$ . In matrix form,

\nabla_{\mathbf{B}}\mathcal{L} = \tfrac2n\,\mathbf{v}\,(\mathbf{X}^\top\mathbf{r})^\top = \tfrac2n\,\mathbf{v}\,\mathbf{r}^\top\mathbf{X} \in \R^{d'\times d}.

Every row of this matrix is a multiple of $\mathbf{r}^\top\mathbf{X}$ , a combination of the rows of $\mathbf{X}$ . So for $\mathbf{u} \perp \mathcal{S}$ we get $\nabla_{\mathbf{B}}\mathcal{L}\,\mathbf{u} = \tfrac2n\mathbf{v}\,\mathbf{r}^\top(\mathbf{X}\mathbf{u}) = \mathbf{0}$ . Each gradient step therefore leaves $\mathbf{B}\mathbf{u}$ unchanged. So does gradient flow, whose derivative $\tfrac{d}{dt}\mathbf{B}\mathbf{u} = -\nabla_{\mathbf{B}}\mathcal{L}\,\mathbf{u}$ is zero. ∎

In plain words, fine-tuning rewrites the features of the kind of inputs it trains on and leaves the features of everything else as they were. Meanwhile the head, which reads all of them, is rewritten. Kumar et al. call this feature distortion. If the ID data spans only part of the input space, the new head is matched to updated features in one subspace and to stale ones in the rest. OOD inputs that live in the rest are then misread.

Their main result, Theorem 3.3 of the paper, lower-bounds the OOD error of fine-tuning in this model under stated assumptions. Their Theorem 3.5 shows that linear probing, which keeps $\mathbf{B}_0$ , does much better OOD when the pretrained features are good. LP-FT avoids the distortion, and their Proposition 3.7 proves it for perfect features. A head that already fits the ID data leaves small residuals $\mathbf{r}$ , so the gradient into $\mathbf{B}$ is small and the features barely move.

✏️ Feature distortion in two dimensions, by hand

Take $d = 2$ and a single feature ( $d' = 1$ ). The pretrained features are perfect, $\mathbf{B}_0 = \mathbf{B}_\star = (c, c)$ with $c = 1/\sqrt2$ , and the true head is $v_\star = 1$ , so the target is $y = c\,x_1 + c\,x_2$ . All ID inputs lie on the $x_1$ axis, so the theorem says fine-tuning cannot change how the feature reads $x_2$ .

Linear probing keeps $\mathbf{B}_0$ and fits $v\,c\,x_1 = c\,x_1$ , so $v = 1$ . That is the exact target, and its OOD error is zero everywhere.

Fine-tuning from $v_0 = 0$ . By the theorem the second feature weight stays at $b_2 = c$ . For the first, write $w = v\,b_1$ , the slope along $x_1$ . The loss depends on $v$ and $b_1$ only through $w$ , so $\partial\mathcal{L}/\partial v = \mathcal{L}'(w)\,b_1$ and $\partial\mathcal{L}/\partial b_1 = \mathcal{L}'(w)\,v$ . Under gradient flow,

\tfrac{d}{dt}\big(v^2 - b_1^2\big) = 2v\,(-\mathcal{L}'b_1) - 2b_1\,(-\mathcal{L}'v) = 0,

so $v^2 - b_1^2$ keeps its initial value $0 - c^2 = -\tfrac12$ . Training ends with the ID data fitted, $v\,b_1 = c$ . Substituting $v^2 = b_1^2 - \tfrac12$ into $v^2b_1^2 = \tfrac12$ gives $b_1^4 - \tfrac12b_1^2 - \tfrac12 = 0$ , whose positive root is $b_1^2 = 1$ . So $b_1 = 1$ and $v = 1/\sqrt2$ . The fine-tuned model is $f(x) = 0.707\,x_1 + v\,b_2\,x_2 = 0.707\,x_1 + 0.5\,x_2$ . It is exact on the ID axis and wrong off it, with squared error $(0.5 - 0.707)^2 = 0.043$ at $x = (0, 1)$ .

LP-FT starts from $v = 1$ , where the ID residuals are zero. Neither parameter moves, and the OOD error stays zero.

The cell runs plain gradient descent on three ID points with a small step and recovers the gradient-flow answer.

Python · cell 4

Gradient descent ends at $v = 0.708$ and $b_1 = 0.999$ . The weight on $x_2$ is 0.5005 where the target has 0.7071, and the OOD error is 0.0427, against the flow's 0.0429.

The same mechanism appears at scale. Kumar et al. tested 10 distribution-shift benchmarks, among them Breeds, DomainNet, and ImageNet-V2, -R, -A, and -Sketch. Fine-tuning beat linear probing by about 2 points ID and lost by about 7 OOD. LP-FT beat fine-tuning by about 1 point ID and 10 OOD (Kumar et al. 2022). This is the catastrophic forgetting of §14.4, the loss of pretrained ability during fine-tuning, seen through the features. A randomly initialized head sends large, poorly aimed gradients into a good encoder at the start of training. In practice, when an encoder will meet shifted data, probe first and then fine-tune, and keep a shifted test set to check. "Why does fine-tuning sometimes lose to a linear probe out of distribution?" is a common interview question, and this mechanism is the answer.

Research

Feature distortion and LP-FT

The setup builds the two-layer linear model at a size where the theorem bites. Inputs live in $\R^{20}$ . There are $d' = 3$ true features, the orthonormal rows of $\mathbf{B}_\star$ , and the pretrained $\mathbf{B}_0$ is a slightly perturbed copy. The 24 ID training inputs span only an 8-dimensional subspace, while OOD inputs fill all of $\R^{20}$ .

Implement transfer(X, y, B0, v0, lr, steps) with full-batch gradient descent on $\mathcal{L} = \tfrac1n\lVert\mathbf{X}\mathbf{B}^\top\mathbf{v} - \mathbf{y}\rVert^2$ . Take both gradients at the current parameters, so the updates are simultaneous. Run three methods, each for steps steps at step size lr.

LP updates only v, starting from (v0, B0).
FT updates both, starting from (v0, B0).
LP-FT runs FT for steps steps, starting from the LP result.

Return {"lp": (v, B), "ft": (v, B), "lpft": (v, B)}. The tests measure ID and OOD errors against the true function and expect the ordering Kumar et al. found.

import numpy as np

def transfer(X, y, B0, v0, lr, steps):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

With the reference solution the ID errors are $1.0\times10^{-2}$ for LP and below $10^{-30}$ for FT and LP-FT. The OOD errors are 0.029 for LP, 0.164 for FT, and 0.019 for LP-FT. Fine-tuning from a random head fits ID best and is more than five times worse than probing OOD. LP-FT is the best on both, the pattern of Kumar et al.'s benchmarks.

Few labels, many tasks, every pixel

A general-purpose encoder will be used on tasks nobody tuned it for, often with few labels. Several families of evaluations try to measure that.

📐 Transfer suites and dense probes

Transfer suites repeat a protocol on many datasets. Kornblith et al. (2019) used 12 classification datasets, with both fixed features and fine-tuning. VTAB, the Visual Task Adaptation Benchmark, has 19 tasks in three groups. They are natural images, specialized ones (medical and satellite), and structured ones (counting, distances, orientations). Its VTAB-1k version gives each task 1,000 labels, 800 for training and 200 for validation. Models are selected on the validation split, retrained on all 1,000, and evaluated on the test set with three seeds, reporting the median (Zhai et al. 2019).

Dense probes train a linear map from every frozen patch token to the task. For semantic segmentation it is a $K$ -way classifier per patch, scored by mIoU, the intersection over union of predicted and true regions averaged over classes (§27.6), on ADE20k, Cityscapes, and VOC. For depth it is a regressor per patch, scored by root mean squared error (RMSE) on NYUv2 and KITTI.

Instance retrieval ranks a database by the cosine similarity of global features to a query. It reports mean average precision (§7.5) on landmark sets such as the revisited Oxford and Paris.

VQA through a language model connects the frozen encoder to a language model, trains the connection, and scores visual question answering. This is the evaluation that matters when the encoder will serve a multimodal model.

Low-shot evaluation adds one more source of noise, the choice of which few images are labeled. The labeled images are called the support set. A low-shot number must therefore be averaged over several draws of the support set and reported with its spread. The cheapest low-shot classifier is the nearest class mean. Average the unit features of each class's support images. Then assign a test image to the class whose average has the largest cosine with it. It needs no training at all.

Core

Low-shot evaluation with honest error bars

Implement low_shot_eval(train_h, train_y, test_h, test_y, n_per_class, n_draws, rng), which returns the mean accuracy over draws and its sample standard deviation over draws (ddof=1), as two floats.

Normalize all features to unit length.
For each of n_draws draws, and for each class c in np.unique(train_y) in increasing order, pick that class's support set with rng.choice(np.flatnonzero(train_y == c), size=n_per_class, replace=False). Use no other random calls.
Average each class's normalized support features and renormalize the means to unit length.
Predict every test image's class as the one whose mean has the largest cosine with it, and record the accuracy of the draw.

The standard deviation is how far one person evaluating one draw should expect to be off. The standard error of the mean is smaller by $\sqrt{\text{n\_draws}}$ and answers a different question.

import numpy as np

def low_shot_eval(train_h, train_y, test_h, test_y, n_per_class, n_draws, rng):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

With the reference solution and np.random.default_rng(3), 30 draws on the exercise's data give 45.8 percent with a standard deviation of 7.5 points at one shot per class. Five shots give 67.7 plus or minus 4.8, and twenty give 79.8 plus or minus 2.2. At one shot, two honest evaluations of the same encoder can easily differ by ten points, so a single draw tells you very little.

The published transfer studies tell one consistent story. No single number ranks representations for every use.

📌 Protocols and tasks disagree

Supervised features (Kornblith et al. 2019, 16 ImageNet networks on 12 datasets). ImageNet accuracy correlates with transfer accuracy at $r = 0.99$ for fixed features and $0.96$ for fine-tuning. But common regularizers (label smoothing, dropout, auxiliary heads) that slightly raise ImageNet accuracy make the penultimate features much worse for transfer.
Augmentation (Chen et al. 2020c, MoCo v2). Adding blur augmentation alone gives a 63.4 linear probe, against 66.2 for adding the projection head alone. Yet blur gives better detection on VOC after fine-tuning (56.8 AP against 56.4). The authors conclude that linear classification accuracy is not monotonically related to transfer performance in detection.
Masked modeling (He et al. 2022). MAE ViT-L loses the linear probe to MoCo v3 and wins at every partial fine-tuning depth.
Self-supervised features (Ericsson et al. 2021, 13 models on 40 tasks). ImageNet top-1 predicts many-shot recognition well, and predicts few-shot recognition, detection, and dense prediction less and less. No single method dominates. The self-supervised models keep less color information than supervised ones, which the authors attribute to their heavy color augmentation.
VTAB (Zhai et al. 2019, a 2019 snapshot with ResNet-50s). Supervised ImageNet pretraining helped most on natural tasks. In the paper's lightweight hyperparameter sweep, adding a rotation-prediction loss to it gave the best model overall (67.5 against 65.6 average on VTAB-1k), mostly through the structured tasks.

Dense probes have their own settings that move the number. Segmentation accuracy depends heavily on resolution, which is why DINOv3 segments with every model at 1,024 patch tokens (448 pixels at patch size 14, 512 at 16). It also depends on the high-norm artifact tokens that large ViTs grow. These corrupt feature maps unless the model has the registers of §27.4, extra tokens that give the model somewhere else to store global information. And a global loss does not guarantee good local features. DINOv3's dense scores degraded during long training while its global ones kept rising. Its fix was Gram anchoring (§28.4), a loss that keeps the patch-to-patch similarity matrix of the student close to that of an earlier checkpoint whose dense features were still good. In practice, a dense evaluation reports its resolution, its token count, and whether registers were removed, or its number cannot be compared.

A simulation in which the rankings flip

Now put everything together on three synthetic "pretrained" encoders for one ten-class task. Each caricatures a family of this chapter.

The clustered features are noisy class centers, the geometry contrastive and DINO training aim for.
The spread features caricature MAE, in three parts. They have a weak linear class signal. They have a strong class signal multiplied by a random sign that flips from image to image. That signal averages to zero over images, so no linear map can read it, while a nonlinear map such as an absolute value can. And they have 32 loud nuisance directions that carry nothing, like the textures a pixel-reconstruction loss must keep.
The collapsed features squeeze the clustered ones into 3 dimensions.

Each is scored by weighted 20-NN, a linear probe, a one-hidden-layer MLP probe (a stand-in for partial fine-tuning), and the effective rank.

For example, singular values $(3, 3, 3, 0)$ give shares $(1/3, 1/3, 1/3, 0)$ , entropy $\log 3$ , and effective rank 3, ignoring the tiny $\epsilon$ . The cell computes it on the centered training features.

Python · cell 5

✏️ Three protocols, three rankings

encoder	20-NN	linear probe	MLP probe	effective rank
clustered	0.874	0.884	0.871	15.5
spread	0.255	0.591	1.000	54.0
collapsed	0.295	0.315	0.315	2.9

By 20-NN the order is clustered, collapsed, spread. The spread encoder beats the 0.1 of chance only through its weak linear part. By the linear probe it is clustered, spread, collapsed. By the MLP probe the spread encoder is first, at 1.000. The effective rank puts spread first, then clustered, then collapsed.

Every protocol told the truth about one property. The spread features hold the most information. The clustered ones hold it in the most accessible form. The collapsed ones have lost some of it for good, and the rank saw that without labels.

Measuring a representation without labels

Every protocol so far needs labels. During pretraining, or on a new domain with no labels, you would like a number that predicts them. The effective rank of the embeddings is such a number. Garrido et al. (2023) proposed it for this job and named it RankMe. They compute it on the embeddings $\mathbf{z}$ , the projection head's output, of 25,600 unlabeled images, and use it to predict how well the representation $\mathbf{h}$ will probe. They take the embedding matrix as it is, without centering. The cell above centered first, so state your convention when you compare numbers.

The evidence is empirical. Across hundreds of runs of joint-embedding methods (the two-branch methods of this chapter), Garrido et al. found that within one method a higher rank goes with a higher linear-probe accuracy, on ImageNet and on other datasets. They then used it to pick hyperparameters such as loss weights, temperatures, learning rates, and weight decay. Choosing the run with the highest rank lost under half a point on average against choosing by labeled ImageNet validation accuracy.

The rank theorem above is the necessary half of the reason, and the simulation shows its limit. The spread encoder had the highest rank and the worst k-NN, because nuisance directions inflated its rank. So compare effective ranks only between runs of the same method, as Garrido et al. advise, and treat the number above all as an alarm for collapse. In practice it sits next to the other label-free checks of §28.3. One is the per-coordinate standard deviation of the unit embeddings, which falls to zero under complete collapse. The other is a plot of the singular-value spectrum, which falls off a cliff under dimensional collapse.

How much a top-1 number moves by chance

Suppose two checkpoints score 81.2 and 80.9 percent on ImageNet's 50,000 validation images. Is the first one better? Two kinds of randomness stand between you and the answer. One is which images happen to be in the test set. The other is the training run itself, whose random seed changes the initialization, the data order, and the augmentations.

The standard error comes from the binomial count of correct answers. Each image is right with probability $p$ , so the count of right answers has variance $np(1-p)$ , and the accuracy has variance $p(1-p)/n$ . On small test sets, replace the normal interval by the Wilson interval of §0.11. It inverts the normal test instead of plugging in the measured accuracy, so it stays inside 0 to 1.

Pairing helps because most images do not distinguish the models. Call an image discordant if exactly one of the two models gets it right. Let $a_i$ and $b_i$ be 1 when models A and B get image $i$ right, and 0 otherwise. Each image contributes a difference $b_i - a_i \in \{-1, 0, 1\}$ . Images both get right or both get wrong contribute zero. The squared difference is 1 on discordant images and 0 elsewhere, so its mean is $\delta$ , the fraction of discordant images. The mean difference is the gain $g$ . One difference therefore has variance $\delta - g^2$ , and their average over $n$ images has variance $(\delta - g^2)/n$ (§0.11 gives the same result). With $\delta = 0.07$ and $n = 50{,}000$ the paired standard error is about $\sqrt{0.07/50{,}000} = 0.12$ points. Treating the two accuracies as independent gives about 0.25.

McNemar's test uses only the discordant images. It asks whether "B right, A wrong" is more common than "A right, B wrong". Under the null hypothesis of no difference, each discordant image is equally likely to go either way, so it is a binomial test with probability one half.

The cell simulates two ImageNet-sized evaluations with a true gain of 0.3 points and 7.3 percent discordant images. It then runs the paired bootstrap of §0.11 in its fastest form. A paired bootstrap resamples the test images with replacement, scores both models on each resample, and reads an interval off the spread of the differences. Resampling only changes how many images fall into each of the four cells (A right or wrong, B right or wrong). So one multinomial draw of the four counts is one bootstrap replicate.

Python · cell 6

The widget replays this example at its true gain of 0.3 points, not the sampled 0.22, and draws the 95 percent interval around it. The interval must clear zero before the gain means anything.

The same arithmetic, for benchmarks of every size, is the subject of the statistics of benchmarks in Chapter 22. The lesson for this chapter is short. When a paper's ablation table moves by 0.2 points, read it as "no measurable difference" unless it reports several seeds.

Which frozen encoder, as of late 2026

Encoder numbers can only be compared on one protocol and one resolution. Here is the frontier measured that way. Three kinds of encoder appear. Self-supervised ones (DINOv2, DINOv3, Web-DINO, Franca) learn from images alone. Image-text ones (SigLIP 2, and PE-core, the core model of Perception Encoder) learn to match images with captions, as CLIP does. Agglomerative ones (AM-RADIO, PE-spatial) are distilled from several teacher models at once.

📌 Frozen encoders compared, as of late 2026

All numbers use frozen backbones, from DINOv3's Tables 3, 7, and 9 (Siméoni et al. 2025). Classification and segmentation run at 1,024 patch tokens. Retrieval resizes each image to 224 pixels on its long side.

encoder	ImageNet linear top-1	ADE20k linear mIoU	NYUv2 depth RMSE (lower is better)	Oxford-Hard retrieval mAP
PE-core G/14	89.3	38.9	0.590	32.7
SigLIP 2 g/16	89.1	42.7	0.494	25.1
DINOv3 7B/16	88.4	55.9	0.309	60.7
AM-RADIO v2.5 g/14	88.0	53.0	0.340	47.5
DINOv2 g/14	87.3	49.5	0.372	58.2
Web-DINO 7B/14	85.9	42.7	0.466	31.2
Franca g/14	84.8	46.3	0.445	14.3

PE-spatial G/14 scores 49.3 mIoU and 0.362 RMSE. On the cleaner ImageNet test sets the top three nearly tie. On ImageNet-V2 they score 81.6, 81.6, and 81.4, and on ReaL 90.4, 90.5, and 90.4. Image-text encoders lead slightly on global classification. The strongest self-supervised ones, DINOv3 and DINOv2, lead by wide margins on dense geometry and instance retrieval. Agglomerative models sit in between.

One controlled comparison changed many opinions. Earlier, self-supervised encoders lagged CLIP inside multimodal models. The usual explanation was that captions supply meaning that images alone cannot. Fan et al. (2025), in a study they call Web-SSL, removed the confound in the data. They trained DINOv2-style and MAE-style models on the images, and CLIP on the image-caption pairs, of the same 2 billion MetaCLIP samples. The DINOv2-style and CLIP models range from 1 to 7 billion parameters, and the MAE-style ones from 1 to 5 billion. They evaluated the frozen encoders by visual question answering (VQA) through a language model, on the 16 tasks of the Cambrian-1 suite.

Visual self-supervision scaled better with model size and data, and it did not saturate at 7 billion parameters. It reached CLIP-level performance on VQA and on classic vision benchmarks. That suggests the earlier gap came largely from the training data, not from language itself. When an encoder will serve a vision-language model, this is the evaluation that counts, and connecting a vision encoder to a language model builds it.

✏️ Choosing a frozen backbone for aerial segmentation

You must segment roads, buildings, and vegetation in satellite tiles. You have 2,000 labeled tiles, and the budget allows only a frozen backbone with a trained probe.

Shortlist from published dense results. DINOv3-class features lead image-text encoders by 13 points or more of ADE20k linear mIoU, a large and consistent dense advantage, so they are the first candidates. But aerial images are far from both ADE20k and the pretraining data, so only your data can decide.
Split by region. Use 1,600 tiles for training and 400 for validation. Split by geographic region, so that neighboring tiles, which look alike, cannot land on both sides.
Probe two or three candidates. Take a self-supervised encoder, an image-text one, and an agglomerative one. Run a linear segmentation probe on the patch tokens at the resolution you will deploy, with registers removed. Try the last block and two intermediate ones.
Put intervals on the result. Report mIoU with a bootstrap over validation tiles, which resamples tiles with replacement. With only 400 tiles, expect intervals of about a point or more, so a half-point win between candidates decides nothing.
Try adaptation last. For the best candidate, compare the frozen probe against LP-FT with a small learning rate. If deployment covers new regions or seasons, keep a held-out region as an OOD test. The feature-distortion result predicts that plain fine-tuning may lose what the probe had there.

Here are the protocols side by side.

protocol	what it trains	what it measures	cost	settings that move it	blind spot
weighted k-NN	nothing	whether neighbors share labels (local geometry)	one pass and a feature bank (2.0 GB for ImageNet at $d = 768$ in float16)	$k$ , $\tau$ , normalization	nonlinear but globally readable structure, nuisance directions
linear probe	$dK + K$ weights	linear readability of one pooled vector	a convex fit, hours on ImageNet	pooling, layer, learning rate, test-tuning	information spread over tokens or nonlinear features
attentive probe	one cross-attention layer, about $3d^2 + dK$	whether some weighting of the tokens holds the class	a little more than a linear probe	heads, which tokens, regularization	it is no longer a linear readout, so it measures the probe too
partial or full fine-tuning	some or all of the encoder	quality as an initialization	a full training run per task	recipe, layer decay, epochs	confounds pretraining with recipe, distorts features out of distribution
LP-FT	probe, then everything	initialization without distortion	a probe plus a run	the same as fine-tuning	features still move if the probed head fits the task poorly
low-shot	any of the above, few labels	label efficiency	cheap per draw	which images are labeled	needs many draws and their spread
transfer suite	any of the above, many tasks	generality	multiplied by the number of tasks	task choice, per-task tuning	averages hide which tasks fail
dense probe	a linear map per patch	local semantics or geometry	a probe on every patch, at high resolution	resolution, registers, layer	global semantics
VQA through a language model	a connector (and maybe the LLM)	usefulness inside a multimodal model	a multimodal training run	the LLM, the data, the connector	the encoder's own geometry
effective rank (RankMe)	nothing, no labels	whether the embedding has collapsed	one SVD	the sample, $\epsilon$	rank is necessary, not sufficient

In PyTorch

The reference loop does everything the definition asks. It freezes the encoder in evaluation mode as well as without gradients, and extracts features under torch.no_grad(). It trains a probe made of a non-affine BatchNorm and a linear layer. It checks that the encoder's entire state, running statistics included, is bit-for-bit unchanged. And it folds the BatchNorm into the linear layer, as the theorem says it can. The last lines show the bug of the pitfall, a single forward pass in training mode moving a running mean.

🔥 In PyTorchfor a machine with PyTorch

It prints a probe accuracy of 0.993, confirms that the encoder's state is unchanged and that the BatchNorm folds, and then shows one training-mode pass moving a running mean by 0.051.

For a real backbone, the same loop wraps a model loaded from a hub. Feature extraction runs once over the dataset and the results are cached to disk, after which the probe trains in minutes. Probing several layers at once costs nothing extra, since one forward pass can return every block's output. That is how DINOv2 searches its grid.

Where evaluations go wrong

These failures are common enough to deserve a checklist.

The benchmark became the objective. ImageNet linear accuracy was the debugging signal of most self-supervised research, so methods are tuned to it. DINOv2 gives this as its reason for also reporting ReaL and V2.
The recipe moved the number. Probe learning-rate schedules, pooling, and layer choice move linear accuracy by points. An early large study found the linear evaluation sensitive to the learning-rate schedule (Kolesnikov et al. 2019). Fine-tuning recipes confound pretraining with training.
k-NN was run on unnormalized features, or with a $k$ tuned on the test set.
The dense probe was run at the wrong resolution, or on feature maps with artifact tokens.
The best layer was not the last. Probe several, as AIM and Perception Encoder show.
The evaluation did not match the use. For an encoder inside a vision-language model, the evaluation that matters is the multimodal model's. Evaluating that way, on matched data, is how Fan et al. showed that self-supervision can match CLIP.
A single run was reported as a difference. The noise section above shows why that is not evidence.

The probing studies of language models in §23.2 meet the same questions from the other side. There a probe is a microscope on a model's internals. Control tasks, which give each word a fixed random label, guard against probes that score well by memorizing instead of reading. Here a probe is a benchmark for an encoder. The same caution applies to attentive and MLP probes, whose extra capacity measures the probe as well as the features.

🧭 What to remember

Each protocol is a question with a budget. k-NN measures local cosine geometry with no training. A linear probe measures linear readability of one pooled vector. An attentive probe lets a cross-attention layer choose the tokens. Partial and full fine-tuning measure the weights as an initialization, and dense probes measure every patch.
A correct linear probe freezes the encoder in evaluation mode as well as without gradients, and states which vector and layer it reads. A non-affine BatchNorm in front of it only changes conditioning. Hyperparameters are selected on held-out training data, never on the test set.
Protocols disagree for real reasons. MAE ViT-H scores 27 points lower by k-NN than by a linear probe, MAE ViT-L loses the linear probe to MoCo v3, and it wins every depth of partial fine-tuning. MoCo v2's blur lowers the linear probe and raises detection. Image-text encoders lead ImageNet probes, while self-supervised DINOv3 leads dense probes by 13 to 17 mIoU as of late 2026.
Fine-tuning from a random head changes features only inside the span of the training data and can lose out of distribution. LP-FT, probe first and then fine-tune, keeps both, about 10 points better OOD than fine-tuning across Kumar et al.'s benchmarks.
On ImageNet's 50,000 images the standard error of top-1 is about 0.18 points and seed noise 0.1 to 0.3, so a single-run gain below half a point is not evidence. Compare models on the same images with paired tests and several seeds.
Without labels, effective rank (RankMe) catches collapse and orders runs of one method. A least-squares probe on a rank- $r$ representation is bounded by Eckart-Young, but high rank is necessary, not sufficient.
These encoders and protocols carry forward. Masked video models are evaluated by fine-tuning and attentive probes on Kinetics and Something-Something. Joint image-text spaces add a protocol that needs no training at all, zero-shot classification. And vision-language models judge an encoder by the answers it helps a language model give. Chapter 29 starts with video.

Kornblith, Shlens & Le (2019). Do Better ImageNet Models Transfer Better?. The fixed-feature and fine-tuning correlations, and the finding that regularizers that help ImageNet hurt transfer.
Zhai et al. (2019). A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark. VTAB's 19 tasks, the VTAB-1k protocol, and a model of honest hyperparameter selection.
Ericsson, Gouk & Hospedales (2021). How Well Do Self-Supervised Models Transfer?. Thirteen self-supervised models on forty tasks, and where ImageNet accuracy stops predicting transfer.
Kumar et al. (2022). Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. Feature distortion proved in linear networks, and LP-FT measured on ten distribution shifts.
Garrido et al. (2023). RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. Effective rank as a label-free criterion, with its Eckart-Young motivation and its limits.
Chen et al. (2022). Context Autoencoder for Self-Supervised Representation Learning. The argument for attentive probing and the table comparing linear, attentive, and fine-tuned evaluations.
Bolya et al. (2025). Perception Encoder: The best visual embeddings are not at the output of the network. A layer-by-layer evaluation showing that the best features can sit deep inside the encoder.
Fan et al. (2025). Scaling Language-Free Visual Representation Learning. Self-supervised and CLIP models trained on the same data and evaluated through a language model, the evaluation that changed the comparison.

Made for larger screens

Evaluating representations

Every protocol is a question with a budget

The linear probe

Frozen means evaluation mode

Which features, and which hyperparameters

A linear probe done right

What a linear probe can and cannot read

The weighted k-NN classifier

Weighted k-NN classifier

Attentive probes

An attentive probe forward pass

Fine-tuning, partial fine-tuning, and LP-FT

Why fine-tuning can hurt out of distribution

Feature distortion and LP-FT

Few labels, many tasks, every pixel

Low-shot evaluation with honest error bars

A simulation in which the rankings flip

Measuring a representation without labels

How much a top-1 number moves by chance

Which frozen encoder, as of late 2026

In PyTorch

Where evaluations go wrong

Read more

Watch next