Chapter 30 · Section 30.3 · 90 min

Zero-shot classification and retrieval

This section is about using an image-text model as a classifier for labels it was never trained on. You type the names of the classes, the model turns each name into a vector, and each image goes to the class whose vector is closest to the image's own vector. No labeled image from the task is used, which is why this is called zero-shot classification. It matters for two reasons. One pretrained model becomes a classifier for any list of labels you can type, and that classifier holds up far better than a supervised one when the images change.

The model is CLIP, a dual encoder. It is a pair of networks, an image encoder and a text encoder, that map photos and sentences into one shared space of vectors (§30.1). It was trained on 400 million web images with their captions. Its only task was matching. In each batch of 32,768 image-caption pairs, every image had to pick out its own caption and every caption its own image (§30.2). It never saw an ImageNet label. Yet its best model, ViT-L/14 at 336 pixels, names the main object in 76.2 percent of ImageNet's 50,000 validation photos, choosing among 1,000 classes. That is the accuracy of the original ResNet-50, a convolutional network trained on ImageNet's 1.28 million labeled images (Radford et al. 2021).

This section turns that one result into a working toolkit. We write the classifier down exactly and find the class prior hidden inside it. We see how the sentences that name the classes change it, and how to fix its overconfidence. We price a class name in labeled images and combine the two. Then we measure image-text retrieval, CLIP's other main use, carry the idea from whole images to boxes around objects, and ask why the classifier is so robust and how to fine-tune it without losing that.

We keep the letters of §30.1. An image is $x$ and a text is $y$ . The image encoder turns $x$ into a vector, and we scale that vector to length one to get the unit vector $\hat{\mathbf u} \in \R^d$ . The text encoder does the same for a text $y$ and gives the unit vector $\hat{\mathbf v}(y) \in \R^d$ . The shared width $d$ is 768 for ViT-L/14, CLIP's largest Vision Transformer, an image encoder that cuts each image into patches of 14 by 14 pixels (§27.1). The cosine of an image and a text is the dot product of their unit vectors, $\hat{\mathbf u}^\top\hat{\mathbf v}$ , a number between $-1$ and $1$ . CLIP multiplies every cosine by a learned positive number $e^{\alpha}$ , the logit scale, before its softmax. In the released models it sits at its upper limit of 100. Vectors are columns, so $\hat{\mathbf u}^\top\hat{\mathbf v}$ is a dot product, and a matrix that stacks embeddings holds one embedding per row.

A zero-shot classifier is a linear head

Here is the recipe before any symbols. Take the task's class names, say dog, cat, and car. Write each name into a few short sentences, such as a photo of a dog. and a drawing of a dog.. These fill-in sentences are called prompt templates. Embed every sentence with the text encoder and scale each embedding to length one. Average the embeddings of each class, and scale the average back to length one. The result is one unit vector per class, called the class's prototype. To classify an image, embed it, take its cosine with every prototype, and pick the largest.

Two more words are needed for the numbers. The scores a classifier feeds into its final softmax are its logits. The softmax turns logits $z_1, \dots, z_K$ into probabilities $e^{z_k}/\sum_j e^{z_j}$ that are positive and add up to one. §1.8 met this recipe briefly at the end of its tour of encoders. A tiny example in two dimensions shows every number.

✏️ Three prototypes, one image, two scales

Take $d = 2$ and three unit prototypes, $\mathbf c_{\text{dog}} = (1, 0)$ , $\mathbf c_{\text{cat}} = (0.6, 0.8)$ , and $\mathbf c_{\text{car}} = (-0.6, 0.8)$ . An image has the unit embedding $\hat{\mathbf u} = (0.8, 0.6)$ . Its cosines with the three prototypes are $0.8$ , $0.48 + 0.48 = 0.96$ , and $-0.48 + 0.48 = 0$ . The largest is the cat's, so the prediction is cat.

Multiply the cosines by CLIP's scale of 100 to get the logits $(80, 96, 0)$ . The softmax gives dog the probability $e^{-16}/(1 + e^{-16} + e^{-96}) \approx 1.1\times10^{-7}$ and gives cat essentially 1. At scale 10 the logits are $(8, 9.6, 0)$ . Since $e^{-1.6} = 0.202$ and $1/(1 + 0.202) = 0.832$ , the probabilities are about $(0.168, 0.832, 0.00006)$ . The prediction is cat at both scales, but the confidence is completely different. Only labeled held-out images can say which confidence is right, and that is the subject of calibration below.

Python · cell 1

The example shows two facts. The prediction depends only on which prototype is closest. The confidence also depends on the scale, and the scale CLIP learned makes nearly every answer look certain. Here is the general statement.

📐 Zero-shot classifier

Suppose the task has $K$ classes. For class $k$ , fill each template in a set $\mathcal T$ (a photo of a {label}., a drawing of a {label}., and so on) with the class name. Call the filled sentence $y_{km}$ for template $m = 1, \dots, \lvert\mathcal T\rvert$ , and write $\hat{\mathbf v}_{km} = \hat{\mathbf v}(y_{km})$ for its unit embedding. The prototype of class $k$ is the normalized mean of its template embeddings,

\mathbf m_k = \frac{1}{\lvert\mathcal T\rvert}\sum_{m=1}^{\lvert\mathcal T\rvert}\hat{\mathbf v}_{km} \in \R^d, \qquad \mathbf c_k = \frac{\mathbf m_k}{\lVert\mathbf m_k\rVert}.

Stack the prototypes as the rows of $\mathbf W \in \R^{K\times d}$ , one row per class. For an image with unit embedding $\hat{\mathbf u} \in \R^{d}$ , the zero-shot logits, probabilities, and prediction are

\mathbf z = e^{\alpha}\,\mathbf W\hat{\mathbf u} \in \R^{K}, \qquad p(k\mid x) = \operatorname{softmax}(\mathbf z)_k, \qquad \hat k = \arg\max_k\, \mathbf c_k^\top\hat{\mathbf u}.

In plain words, the text encoder writes one unit vector per class. The score of a class is the cosine between the image and that vector, times the logit scale. The predicted class is the closest prototype. Nothing is trained.

Now look at the form of $\mathbf z$ . A linear head is the last layer of a classifier, a matrix that turns a feature vector into one score per class. A linear probe is a linear head trained with labels on top of a frozen encoder, one whose own weights stay fixed (§28.6). The zero-shot logits $\mathbf z = e^{\alpha}\mathbf W\hat{\mathbf u}$ are exactly a linear head on the frozen image features, with weight matrix $e^{\alpha}\mathbf W \in \R^{K\times d}$ . CLIP's authors describe it the same way. It is a multinomial logistic regression, and the text encoder writes its weights.

Three things set this head apart from a trained probe. Every class's weight vector has the same length, $e^{\alpha}$ . There is no bias vector. And the weights come from sentences, not from labeled images. Each of the three has a consequence, and together they organize the first half of this section.

The prediction depends only on the directions $\mathbf c_k$ , so the sentences that set those directions matter.
The probabilities also depend on the scale, so the scale must be calibrated.
A preference for one class over another would need a bias, which the head does not have. So what prior does it assume?

We start with the last question. An interviewer who asks why zero-shot CLIP is a linear classifier wants this picture, a head with no bias and equal-length rows, written by the text encoder.

The class prior the head assumes

A classifier's class prior is how likely it considers each class before it looks at the image. A head without a bias seems to have no prior. It does have one, hidden in what its scores were trained to estimate. A small example shows why that matters.

Which prior does the zero-shot head use? It depends on what CLIP's score estimates. §30.1 proved that a perfectly trained CLIP scores a pair by its pointwise mutual information plus one constant, $s(x, y) = \operatorname{PMI}(x, y) + c_0$ . Here $\operatorname{PMI}(x, y) = \log\frac{p(x, y)}{p(x)p(y)}$ , and $p$ is the distribution of image-caption pairs on the web. PMI measures how much more often an image and a caption occur together than they would if they were unrelated. Put the class prompts into that score.

Proof. $\operatorname{PMI}(x, y_k) = \log p(x\mid y_k) - \log p(x)$ , so $e^{s(x, y_k)} = e^{c_0}\,p(x\mid y_k)/p(x)$ . The factor $e^{c_0}/p(x)$ is the same for every $k$ , so it cancels in the softmax over classes. Adding $\log\pi_k$ multiplies the $k$ -th numerator by $\pi_k$ , which is Bayes' rule. $\square$

In plain words, PMI divides out how often each caption occurs on the web. So the ideal zero-shot classifier compares only how well the image fits each class. For the shelter photo it says 2/3 for dog, however common dogs are on the web and however rare they are in the shelter. If you know the task's class frequencies, add their logarithms to the logits.

The cell checks both identities exactly in a small discrete world with three classes and six kinds of image.

Python · cell 2

Both identities hold to machine precision. The web prior of $(0.6, 0.3, 0.1)$ has no effect on the zero-shot probabilities. On a task whose prior is $(0.2, 0.3, 0.5)$ , adding the log prior raises accuracy from 0.570 to 0.649. That is the Bayes rate for this world, the best accuracy any classifier can reach on it.

In practice, when a deployment's class frequencies are known and skewed, add $\log\pi_k$ to the logits. An interviewer who asks for the bias of CLIP's zero-shot classifier wants this answer, a uniform prior at the optimum, plus the caveat in the box below.

Prompts are part of the classifier

The weights of this head are sentences, so writing the sentences is part of building the classifier. Radford et al. (2021) report four effects.

Templates. Web captions are rarely a single word, so a bare class name looks unlike the text the encoder was trained on. The single template A photo of a {label}. raised ImageNet accuracy by 1.3 points over the bare name.
Context. Naming the domain helps on fine-grained and unusual data. CLIP used a photo of a {label}, a type of pet. for the Oxford-IIIT Pets dataset, and the same with "a type of food" and "a type of aircraft" for food and aircraft datasets. It used a satellite photo of a {label}. for satellite images. For datasets that ask the model to read text in the image, putting quotes around the text or number helped.
Words with two meanings. A word with two meanings confuses the text encoder. ImageNet contains both the construction crane and the bird called a crane. In Oxford-IIIT Pets, "boxer" is a dog breed, which a text encoder without context could read as an athlete.
Class names. The prompt-engineering notebook released with CLIP renames classes such as "crane" to "construction crane" and "crane bird", "nail" to "metal nail", and "kite" to "kite (bird of prey)". It reports that the edited names add 1.5 points of top-1 accuracy for ViT-B/32. The edits, and the choice of 80 templates, were found by trial and error on ImageNet's labeled training set.

The last point changes what "zero-shot" means for ImageNet. No training step used ImageNet's labels, but people did look at them while choosing the words. So CLIP's ImageNet number is zero-shot in the sense that no gradient step saw the labels, but not in the sense that no one used them.

Words with two meanings are worth working by hand, because they fail in a way that no amount of image quality can fix. If two classes are both written "a photo of a crane.", their prototypes are the same vector. Their logits are then equal for every image. The argmax breaks the tie the same way every time, so one of the two classes is never predicted. Its recall, the fraction of its images labeled correctly, is exactly zero. The only fix is in the text.

Averaging many prompts in embedding space

CLIP averages the embeddings of 80 templates for each ImageNet class, which is called a prompt ensemble. The gains are large for something that costs nothing at test time.

The average is taken in embedding space. You normalize each template's embedding, average them, and normalize the average, which gives the prototype of the definition. The alternative would be to run 80 separate classifiers and average their probabilities. The embedding-space ensemble caches one prototype per class, so classifying an image costs the same as with a single prompt. How different is it from averaging the 80 classifiers' logits? A small example comes first.

Python · cell 3

The division by the norm in the example holds in general.

Proof. The dot product is linear, so $\mathbf m_k^\top\hat{\mathbf u}$ is the mean of the template cosines. Dividing $\mathbf m_k$ by its norm divides the dot product by the same number. The bound is the triangle inequality, $\lVert\mathbf m_k\rVert \le \frac{1}{\lvert\mathcal T\rvert}\sum_m\lVert\hat{\mathbf v}_{km}\rVert = 1$ , with equality only when all templates of class $k$ embed to the same point. $\square$

In plain words, the ensemble's logit for a class is the average of its template logits, multiplied by a factor $1/\lVert\mathbf m_k\rVert \ge 1$ that belongs to that class. The factor is larger when the class's templates disagree. So the ensemble is not a probability average, in which one very confident template can outvote the rest. It is not a plain logit average either. The final normalization puts every prototype back on the sphere, so a class whose templates scatter is not penalized for having a short mean vector. An exercise in §1.8 checks this second normalization.

In practice, ensemble in embedding space and cache one prototype per class. An interviewer who asks why wants the cost argument (one dot product per class, however many templates) and this identity.

Hand-written templates do not scale to thousands of datasets. The box below shows that a few well-chosen templates do as well as all 80, and it describes two newer ideas. One has a language model write the sentences, and the other learns context vectors from a few labels.

🔬 Prompts written by a language model, and prompts learned from data

After the 80 templates were fixed, the authors of CLIP's notebook also ran a greedy search that kept adding whichever template helped most. It stopped at 7 templates, such as itap of a {}. and a origami {}., and these 7 did a little better than all 80, especially for smaller models. Later detectors reuse these "7 best" templates.

The first newer idea asks a large language model to describe each class, with questions such as "What does a platypus look like?". The generated sentences become that class's templates, averaged as above (Pratt et al. 2023, a method called CuPL). With ViT-L/14 this raised ImageNet top-1 accuracy from 75.54 to 76.69 percent, with no labeled data and far fewer hand-written sentences. It helped most on textures, by 6.5 points. The language model adds knowledge the class name lacks, such as that a platypus looks like a beaver with a duck's bill.

The second learns the context words from data (Zhou et al. 2022a, a method called CoOp). The template becomes $n_p$ trainable vectors followed by the token embeddings of the class name. The vectors are trained by gradient descent through the frozen text encoder, on a few labeled images per class. That is prompt tuning, the parameter-efficient method of §14.6 that learns a few input vectors and changes no weights, applied to the text encoder. Across 11 datasets, CoOp with 16 labeled images per class beat hand-crafted prompts by about 15 points on average.

The price is generality. The learned context transfers poorly to classes outside its training set, even within the same dataset. The same authors reduced the problem by making the context depend on the image (Zhou et al. 2022b, CoCoOp). A learned prompt is a small fine-tuned model, and it can overfit like one.

Calibrating the logit scale

A classifier is calibrated when its confidence matches its accuracy. Among the images it labels with 80 percent confidence, about 80 percent should be right. CLIP's scale $e^{\alpha} = 100$ was learned for a different job, picking the right caption out of 32,768 in a training batch. Nothing makes it right for a 10-way or a 1,000-way classification. At test time there are fewer candidates, the prompts are not real captions, and the cosines between images and prompts are bunched in a narrow range. LeVine et al. (2023) found zero-shot CLIP miscalibrated across models, datasets, and prompts. They also found that one temperature fitted per model carries over to other datasets and prompts.

The fix is temperature scaling (§0.11). Replace $e^{\alpha}$ by a scale $\gamma > 0$ fitted on held-out labeled images. (The name comes from writing the scale as one over a temperature.) Fit $\gamma$ by minimizing the negative log-likelihood (NLL), the average over images of $-\log p(\text{true class}\mid x)$ . Multiplying all logits by a positive number never changes their order. So the predictions and the accuracy stay exactly the same, and only the confidences move.

That is also why the fit must use the NLL. Accuracy is the same at every positive scale, so an optimizer told to maximize accuracy has nothing to work with. The NLL is a proper scoring rule, a score whose expected value is best when the predicted probabilities equal the true ones. In the three-prototype example, scales 100 and 10 predict the same class, and only the held-out NLL can say which confidence to believe.

Two tools from §0.11 show calibration. A reliability diagram sorts the images by their top-1 confidence into bins and draws the accuracy within each bin, so a calibrated model has bars on the diagonal.

In plain words, the ECE is the gap between confidence and accuracy in each bin, averaged with weights proportional to how many images the bin holds. The widget shows both tools on synthetic cosines whose labels were drawn at a known scale. Slide the scale and watch the bars, the ECE, and the NLL move, while the accuracy never changes.

Core

Calibrate a zero-shot classifier

You are given zero-shot cosines cos of shape (N, K), one row per image and one column per class prototype, and true labels labels of shape (N,). Write two functions.

zs_nll(scale, cos, labels) returns the mean over images of $-\log\operatorname{softmax}(\gamma\cdot\cos)_{\text{label}}$ , with $\gamma$ = scale. It must stay finite for scales up to 1000.
fit_logit_scale(cos, labels, lo=1.0, hi=1000.0) returns the scale in $[\text{lo}, \text{hi}]$ that minimizes zs_nll, found by a bounded one-dimensional search. scipy.optimize.minimize_scalar with method="bounded" is fine.

The setup gives you ece_top1(probs, labels, n_bins=15), the expected calibration error defined above. The tests use it to check that calibration helps.

import numpy as np

def zs_nll(scale, cos, labels):
    raise NotImplementedError

def fit_logit_scale(cos, labels, lo=1.0, hi=1000.0):
    return 100.0          # placeholder: CLIP's training scale

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Thousands of classes, in chunks

A zero-shot classifier costs one matrix product per batch of images. The product is cheap next to the image encoder. With a large label set, though, the matrix of scores it produces can be too big to hold in memory at once.

✏️ The cost of a zero-shot head at real scale

For ImageNet's 1,000 classes and $d = 768$ , the head is $1{,}000\times768$ , about 0.77 million multiply-adds per image, or 1.5 million floating-point operations. The ViT-L/14 image encoder has about 304 million weights and, at 224 pixels, reads 257 tokens (256 patches plus one class token). At about two floating-point operations per weight per token, one image costs about $2\times304\text{M}\times257 \approx 156$ billion operations. The head adds about one part in a hundred thousand. Building it costs $1{,}000\times80 = 80{,}000$ text-encoder passes, once.

ImageNet-21k has 21,841 classes. Scoring 50,000 images against all of them at once in float32 means a $50{,}000\times21{,}841$ matrix of $1.09\times10^9$ entries, which takes 4.4 GB. The prototype matrix itself, at $d = 768$ in float16, is only $21{,}841\times768\times2$ bytes, or 34 MB. Scoring 256 images at a time keeps the live block of scores at $256\times21{,}841\times4$ bytes, or 22 MB.

Production evaluation code works in blocks like this. Interviewers like the task because it tests top- $k$ logic and the boundary of the last block at once. Top- $k$ accuracy is the fraction of images whose true class is among the $k$ highest-scoring classes, and top-1 accuracy is the ordinary accuracy. Retrieval over millions of items replaces the exhaustive product by an approximate nearest-neighbor index (§7.4).

Interview

Top-1 and top-5 accuracy over a large label set, in chunks

Write chunked_topk_accuracy(img, protos, labels, k=5, chunk=256). The inputs are unit image embeddings img of shape (N, d), unit prototypes protos of shape (C, d), and integer labels of shape (N,). Return (top1, topk), the fraction of images whose label is the highest-scoring class and the fraction whose label is among the k highest.

Compute scores only through the provided score_block(img_block, protos), one block of at most chunk images at a time, so that no more than chunk x C scores ever exist at once. $N$ need not be a multiple of chunk.

import numpy as np

def chunked_topk_accuracy(img, protos, labels, k=5, chunk=256):
    S = score_block(img, protos)                  # placeholder: the whole (N, C) matrix at once
    return float((S.argmax(1) == labels).mean()), 0.0

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

How many labeled images is a class name worth

A k-shot classifier is trained on $k$ labeled images per class. The natural rival of zero-shot is a few-shot linear probe on the same frozen features. Both are linear heads, and only the source of their weights differs. Radford et al. (2021) ran this comparison at 1, 2, 4, 8, and 16 shots. They fitted logistic regression with an L2 penalty, a cost on the squared size of the weights, on CLIP's own features.

📌 Zero-shot is worth about four labeled examples per class

Averaged over the 20 datasets of CLIP's suite that have at least 16 examples per class, zero-shot CLIP matches a 4-shot logistic-regression probe on the same features. So 1-shot and 2-shot probes are worse than the class name alone. On ImageNet, zero-shot matches a 16-shot probe. The number of shots needed to match zero-shot ranges from under 1 (Flowers102 and EuroSAT, where zero-shot loses even to a 1-shot probe) to an estimated 184 per class, with a median of 5.4 and a mean of 20.8.

Against a fully supervised probe on ResNet-50 features, zero-shot CLIP wins on 16 of 27 datasets, including ImageNet. A fully supervised probe on CLIP's own features beats zero-shot CLIP by 10 to 25 points on most datasets (Radford et al. 2021).

Why is one labeled image worth less than a class name? Because an image shows many things at once. A single photo of a boxer shows a dog, a lawn, a collar, a person's legs, and a certain light. A probe trained on it cannot tell which of these defines the class. Many explanations fit one example, so the probe needs several examples to average away everything except the concept. The class name says directly which concept is meant.

With enough images the probe wins. The prototype is one fixed guess at the class direction. The probe fits whatever direction actually separates the task's images, including features no caption ever names.

The simulation below makes this concrete in a world we control. Twenty classes have true unit directions in $\R^{64}$ . Images are noisy unit vectors around them. The text prototypes point roughly the right way, each off by a fixed amount of error. We compare four classifiers.

Zero-shot uses the prototypes.
A plain probe learns from the labeled shots alone.
A text-prior probe starts at the zero-shot head and is pulled back toward it by a penalty.
A class mean plus text adds two copies of each class's prototype to the sum of its labeled shots, a closed form with no training.

Python · cell 4

Zero-shot scores 0.663, averaged over four random worlds. The plain probe scores 0.403 with 1 shot and 0.595 with 2. It passes zero-shot between 2 and 4 shots (0.744 at 4), close to CLIP's real crossover. The text-prior probe beats both at 1, 2, and 4 shots (0.698, 0.738, 0.780). By 8 shots the plain probe overtakes it (0.861 against 0.838). The reason is that the penalty still pulls toward a prototype that is wrong by a fixed amount. A smaller penalty, chosen on held-out data, would track the better of the two curves. The class mean plus text, which counts the class name as two extra examples, is best at every budget (0.738 at 1 shot, 0.934 at 16).

Do not read that ranking as general. In this world the noise is the same in every direction, so the nearest class mean is close to the best possible rule. Real features have correlated nuisance directions, which a logistic probe can learn to ignore and a class mean cannot. The lesson is the shape of the curves. The text is worth a few shots, and the right way to use it is as a prior whose weight shrinks as labeled data arrives.

The widget runs a smaller version of this world, with the class mean as the probe. Slide the weight $\lambda$ given to the text and the error $\eta$ of the text prototypes.

Combining the text with the shots

The principled way to combine them treats the text as a prior belief about the weights. Let the probe have weights $\mathbf W_{\text{p}} \in \R^{K\times d}$ , one row per class, with logits $\mathbf W_{\text{p}}\hat{\mathbf u}$ laid out like the zero-shot head. Take the scaled zero-shot head $\mathbf W_0 = e^{\alpha}\mathbf W \in \R^{K\times d}$ as the prior mean of $\mathbf W_{\text{p}}$ , and put a Gaussian prior around it. Given $N$ labeled examples $(\hat{\mathbf u}_i, k_i)$ , the most probable weights after seeing the data, the maximum a posteriori (MAP) estimate, are

\hat{\mathbf W}_{\text{p}} = \arg\min_{\mathbf W_{\text{p}} \in \R^{K\times d}}\; \frac{1}{N}\sum_{i=1}^{N} -\log\operatorname{softmax}(\mathbf W_{\text{p}}\hat{\mathbf u}_i)_{k_i} + \frac{\lambda}{2}\lVert\mathbf W_{\text{p}} - \mathbf W_0\rVert_F^2 .

In plain words, fit the labels, but pay for every step away from what the text said. As $\lambda \to \infty$ this is the zero-shot head, and at $\lambda = 0$ it is a plain probe. The data term is an average over $N$ examples, so the prior's weight relative to the data should shrink like $1/N$ . For a Gaussian prior of variance $\sigma_0^2$ it is exactly $\lambda = 1/(N\sigma_0^2)$ , which is why the cell used $\lambda = 0.3/N$ .

In practice, tune $\lambda$ on held-out data for each label budget, and keep the logit scale inside $\mathbf W_0$ . The box below explains both and gives a training-free alternative.

Research

A probe that starts from the text

Write two functions. Here X is (N, d) unit image features, y is (N,) labels, and protos is (K, d) unit prototypes.

zs_probe_loss_grad(W, X, y, W0, lam) returns the MAP loss above and its gradient with respect to W, which holds $\mathbf W_{\text{p}}$ . The loss is the mean cross-entropy of the logits X @ W.T plus $\frac{\lambda}{2}\lVert\mathbf W_{\text{p}} - \mathbf W_0\rVert_F^2$ .
fit_zs_probe(X, y, protos, scale, lam, steps=300, lr=1.0) sets $\mathbf W_0$ = scale * protos, starts at $\mathbf W_0$ , takes steps plain gradient-descent steps of size lr, and returns W of shape (K, d).

The tests check the gradient by finite differences. They also check that a huge lam returns the zero-shot head, that lam = 0 is plain descent from $\mathbf W_0$ , and that on seeded 2-shot tasks the result beats both zero-shot and a plain probe.

import numpy as np

def zs_probe_loss_grad(W, X, y, W0, lam):
    raise NotImplementedError

def fit_zs_probe(X, y, protos, scale, lam, steps=300, lr=1.0):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The table collects the ways to adapt a zero-shot head, from no labels to many. The last column previews the robustness results at the end of this section. There, effective robustness (ER) is defined as the accuracy a model keeps under a change of images beyond what its ordinary accuracy predicts.

method	labels needed	what is trained	cost at test	robustness under shift, as measured
zero-shot head	none	nothing	$K$ dot products	the reference point, high ER
prompt ensemble, or prompts written by a language model	none	nothing	same as one prompt	zero-shot CLIP's numbers already use ensembles
linear probe	many per class	$K\times d$ head	same	ER fades as labels grow (Radford et al. 2021, Figures 14 and 15)
probe pulled toward the text	a few per class	$K\times d$ head	same	between zero-shot and the probe, set by $\lambda$
cache of the shots (Tip-Adapter)	a few per class	nothing, optionally the cache	a lookup over the cached shots	not studied in this section
learned context vectors (CoOp)	a few per class	$n_p\times d$ context vectors	one cached text pass per class	above a linear probe on ImageNet shifts (Zhou et al. 2022a)
full fine-tuning	many	everything	same	often below zero-shot (Wortsman et al. 2022)
averaging weights with the zero-shot model, or fine-tuning with the contrastive loss	many	everything	same	keeps most of it, see below

Image-text retrieval, measured correctly

Image-text retrieval ranks a collection of captions for an image, or a collection of images for a caption. The item you search with is the query, and the collection you search is the gallery. Retrieval is the task CLIP was trained for, so it is the first check of any new contrastive model. Two test sets dominate. Flickr30k's standard test split has 1,000 images with 5 human captions each. The COCO 5K test split has 5,000 images and 25,000 captions. Both are scored in two directions, and because every image has five captions, the two directions are not symmetric.

📐 Recall at K for image-text retrieval with several captions per image

Let images $i = 1, \dots, N$ and captions $j = 1, \dots, N_T$ have unit embeddings $\hat{\mathbf u}_i$ and $\hat{\mathbf v}_j$ , and scores $s_{ij} = \hat{\mathbf u}_i^\top\hat{\mathbf v}_j$ . The logit scale is dropped, since a positive factor never changes a ranking. Let $o(j)$ be the image that caption $j$ describes, and $\mathcal C_i = \{j : o(j) = i\}$ the captions of image $i$ . Ties count against the model.

Image to text (CLIP calls it text retrieval). The rank of image $i$ is $r_i = 1 + \lvert\{j \notin \mathcal C_i : s_{ij} \ge \max_{j'\in\mathcal C_i}s_{ij'}\}\rvert$ , the position of its best caption among all captions, and $\operatorname{R@}K = \frac{1}{N}\sum_i\mathbb 1[r_i \le K]$ .
Text to image (image retrieval). The rank of caption $j$ is $r'_j = 1 + \lvert\{i \ne o(j) : s_{ij} \ge s_{o(j)j}\}\rvert$ , and $\operatorname{R@}K = \frac{1}{N_T}\sum_j\mathbb 1[r'_j \le K]$ .

In plain words, an image query is a hit if any of its five captions makes the top $K$ , and a caption query is a hit if its one image makes the top $K$ . For example, if an image's five captions land at ranks 3, 7, 12, 40, and 41, the image is a hit at $K = 5$ and a miss at $K = 1$ . This "recall" is what §7.5 calls the hit rate at $K$ . It is not that section's recall@k, which would ask what fraction of the five captions were found. Image to text is the easier direction, even though it searches a gallery five times larger, because an image has five chances and a caption has one. Every published table shows the gap.

✏️ Zero-shot retrieval at real scale

model	Flickr30k I→T R@1	Flickr30k T→I R@1	COCO I→T R@1	COCO T→I R@1
CLIP, zero-shot (Radford et al. 2021, Table 13)	88.0	68.7	58.4	37.8
SigLIP 2 g/16 at 384 px, zero-shot (Tschannen et al. 2025, Table 1)	95.4	86.0	72.8	56.1

CLIP's R@5 and R@10 on Flickr30k image to text are 98.7 and 99.4. That benchmark is nearly saturated at $K \ge 5$ , and only R@1 still separates models. CLIP prepends "a photo of" to every caption, which it reports is worth 1 to 2 points of R@1. Retrieval numbers depend on preprocessing like this, which tables rarely list.

How much of a difference between two models is real? Each R@1 is a proportion, the fraction of queries that hit, so its sampling error follows from §0.11. The Wilson interval is a confidence interval for a proportion that stays accurate near 0 and 1, where the textbook normal interval fails. The cell computes it for the numbers above and for ImageNet.

Python · cell 5

To compare two models evaluated on the same queries, use a paired test (§0.11). McNemar's test looks only at the queries where exactly one of the two models succeeds. A paired bootstrap resamples the queries many times and recomputes the difference each time. Either way, the noise in a difference comes only from the queries on which the models disagree, often far fewer than all of them.

Interview

Recall at K for image-text retrieval with five captions per image

Write retrieval_recall(img, txt, txt2img, ks=(1, 5, 10)). The inputs are image embeddings img of shape (N, d), caption embeddings txt of shape (N_T, d), neither normalized, and txt2img of shape (N_T,), the image index of each caption. An image may have any number of captions, at least one.

Return (i2t, t2i), two dicts that map each K in ks to the recall defined above. Rank by cosine. Count ties against the model, so that a wrong item scoring exactly as high as the right one pushes it down a rank.

import numpy as np

def retrieval_recall(img, txt, txt2img, ks=(1, 5, 10)):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Hubs, and normalizing over the queries

Per-query recall hides a failure that shows up when you count from the gallery's side. Rank the gallery for every query, and tally how often each gallery item comes first. In high-dimensional embedding spaces the tally is badly skewed. A few items win a large share of all queries, while most items are never the top answer to anything (Radovanović et al. 2010). Those few items are called hubs. Every query a hub wins wrongly is a miss. §29.5 met hubs among video clips and corrected them with querybank normalization. Here we measure hubness exactly and prove why that correction works.

📐 k-occurrence and hubness

Let queries $q = 1, \dots, Q$ and gallery items $j = 1, \dots, G$ have scores $s_{qj}$ . The k-occurrence of item $j$ is the number of queries that rank it among their top $k$ ,

N_k(j) = \big\lvert\{q : j \text{ is among the } k \text{ highest-scoring gallery items for } q\}\big\rvert .

Each query contributes exactly $k$ items, so $\sum_{j=1}^{G} N_k(j) = kQ$ and the mean k-occurrence is $kQ/G$ . A hub is an item whose $N_k$ is far above that mean, and an anti-hub is one with $N_k = 0$ . Hubness is measured by the skewness of $N_k$ over the gallery, $\mathbb E\big[(N_k - \bar N_k)^3\big]/\operatorname{sd}(N_k)^3$ , where $\bar N_k$ and $\operatorname{sd}(N_k)$ are the mean and standard deviation of $N_k$ over the items. The skewness is 0 for a symmetric distribution, and large and positive when a few hubs collect many queries.

Where do hubs come from in a dual encoder? Two facts about trained embeddings combine. Each encoder maps its inputs into a narrow cone, a small region of the sphere (§30.6). And trained embeddings are anisotropic, with a few directions shared by nearly every vector (§1.9). Suppose a gallery item leans further than its neighbors along a direction that all queries share. It then gains about the same extra score against every query, as if it carried a private bonus. Generic captions and visually bland images behave this way.

The cell builds that situation on purpose. Each of 600 images and its caption show one random concept. Every caption leans along one shared direction, and the images lean along it by different amounts.

Python · cell 6

The mean 1-occurrence is exactly 1. Yet one image is ranked first by 39 captions, two thirds of the images are never ranked first, and the top 1 percent of images take 22 percent of all queries. Raw text-to-image R@1 is 0.282. The last three lines of the output apply the standard fix, which needs a definition.

📐 Inverted softmax

Take a bank $\mathcal B$ of queries, a sample of the queries the system might receive, with scores $p_{bj}$ of bank query $b \in \mathcal B$ against gallery item $j$ . Take an inverse temperature $\beta > 0$ . The inverted-softmax score of item $j$ for query $q$ is

\eta_{qj} = \frac{\exp(\beta s_{qj})}{\sum_{b\in\mathcal B}\exp(\beta p_{bj})},

and each query ranks the gallery by $\eta_{qj}$ , in practice by $\log\eta_{qj} = \beta s_{qj} - \log\sum_{b\in\mathcal B} \exp(\beta p_{bj})$ . When the bank is the set of test queries, every column of $\eta$ sums to 1. It is then a softmax over the queries for each gallery item, the transpose of the usual softmax.

In plain words, before an item may claim a query, ask how strongly it claims every query. A hub has a large denominator, because it scores well against everyone, so it is demoted. An item that scores well against this query alone keeps its rank. The exact statement is short.

Proof. $\beta(s_{qj} + \Delta_j) - \log\sum_{b}\exp\big(\beta(p_{bj} + \Delta_j)\big) = \beta s_{qj} + \beta\Delta_j - \beta\Delta_j - \log\sum_{b}\exp(\beta p_{bj})$ , because the factor $e^{\beta\Delta_j}$ comes out of every term of the sum. For the second claim, a softmax over the gallery exponentiates row $q$ and divides it by one positive number, and both steps preserve the order of its entries. $\square$

The second claim is the classic bug, and the cell's last line shows it. Normalizing along the wrong axis leaves R@1 at exactly 0.282, while a bank of 2,000 other captions lifts it to 0.705. The toy was built so that a per-item lean, nearly an additive offset, is its main source of error, so its gain is far larger than real data shows. On real benchmarks the gains are a few points. With the official CLIP ViT-B/32 on COCO 5K text to image, normalizing with a bank of training captions raised R@1 from 30.3 to 34.8 (Bogolin et al. 2022, Table 10). That paper calls the method querybank normalization.

Core

Find the hubs and normalize them away

Write two functions on a score matrix S of shape (Q, G), queries by gallery items.

k_occurrence(S, k) returns the integer array N of shape (G,), where N[j] counts the queries that have item j among their k highest scores.
inverted_softmax(S, beta, bank=None) returns the (Q, G) matrix of log-scores $\beta s_{qj} - \log\sum_b\exp(\beta p_{bj})$ . Here bank is a (n_bank, G) matrix of bank-query scores, and bank=None means the bank is S itself. The result must stay finite for beta up to 1000 with cosines near 1.

import numpy as np

def k_occurrence(S, k):
    raise NotImplementedError

def inverted_softmax(S, beta, bank=None):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

From images to regions

Everything so far scores a whole image. Many uses need to know where things are. A robot is asked for the red mug, a photo archive is searched for a dog on a skateboard, and a safety audit counts forklifts. §27.6 built detectors and segmenters whose classification heads end in a learned weight matrix with one row per class. That is the same fixed list of labels that §30.1 set out to remove. The zero-shot idea removes it here too. Replace the detector's learned class weights by text embeddings of the queries, and the list of classes becomes something you type at test time. A detector that works this way is called open-vocabulary. The first obstacle is that a CLIP image encoder is not built to say where anything is.

Why CLIP's patch tokens are poorly localized

CLIP's loss reads one vector per image. In a CLIP ViT that vector is the final state of the class token, an extra token whose output summarizes the image, after a LayerNorm and the projection $\mathbf W_I$ into the shared space. In CLIP's ResNets it is the output of an attention-pooling layer over the last feature map (§30.2). So the loss reaches the image encoder only through the pooled vector.

In a ViT, the final layer's outputs at the patch positions feed nothing, so they receive no gradient at all. §31.1 uses the same argument to explain why vision-language models read the second-to-last layer. Earlier patch tokens are trained only for what they pass on, through attention, to the class token. Nothing asks the token at position $i$ to describe what is at position $i$ rather than some mix of the whole image. Whatever the loss does not ask for, it does not get.

CLIP's ResNets show what a single location contributes. Their pooling layer is one attention step whose only query is the mean feature of the map. Its output is an attention-weighted sum of per-location values, passed through a final linear map.

∴ Attention pooling averages per-location embeddings

Let a single-head attention-pooling layer have per-location values $\boldsymbol\nu_i \in \R^{d_v}$ for $i = 1, \dots, n$ (written $\boldsymbol\nu$ so as not to clash with the text embeddings $\mathbf v$ ). Let its attention weights be $a_i \ge 0$ with $\sum_i a_i = 1$ , a softmax of the mean query against the keys, and let its output map be affine, $F(\mathbf x) = \mathbf W_O\mathbf x + \boldsymbol\rho$ with $\mathbf W_O \in \R^{d\times d_v}$ and $\boldsymbol\rho \in \R^{d}$ . Then the pooled embedding is

F\Big(\sum_{i=1}^{n} a_i\boldsymbol\nu_i\Big) = \sum_{i=1}^{n} a_i\,F(\boldsymbol\nu_i),

a convex combination of the per-location embeddings $F(\boldsymbol\nu_i)$ , each of which lives in the joint space.

Proof. $F(\sum_i a_i\boldsymbol\nu_i) = \mathbf W_O(\sum_i a_i\boldsymbol\nu_i) + \boldsymbol\rho = \sum_i a_i(\mathbf W_O\boldsymbol\nu_i + \boldsymbol\rho)$ , using $\sum_i a_i = 1$ for the bias. $\square$

∂ The same identity with several heads

With several heads, each head $h$ has its own weights $a_i^{(h)}$ , values $\boldsymbol\nu_i^{(h)}$ , and block of columns $\mathbf W_O^{(h)}$ of the output matrix. The pooled embedding is

\boldsymbol\rho + \sum_h\sum_i a_i^{(h)}\mathbf W_O^{(h)}\boldsymbol\nu_i^{(h)},

a convex combination over locations within each head. The per-location embedding $F(\boldsymbol\nu_i) = \boldsymbol\rho + \sum_h\mathbf W_O^{(h)}\boldsymbol\nu_i^{(h)}$ collects the same per-head terms without the attention weights. If every head attends to location $i$ alone, the pooled embedding is exactly $F(\boldsymbol\nu_i)$ .

In plain words, the image embedding the loss trained is an average of one embedding per location. So each $F(\boldsymbol\nu_i)$ is already a vector that can be compared with text. The attention outputs at individual locations were never trained, because only the mean query was.

Reading these per-location embeddings against the prompts of this section gives a segmenter that needs no training at all. It works, but it is still far from a trained segmenter. The box below gives the recipe and its numbers.

🔬 A segmenter from CLIP with no training

A training-free segmenter uses exactly this (Zhou et al. 2022c, a method called MaskCLIP). It drops the query and key of CLIP's last attention layer. It turns the value map and the final linear map into two $1\times1$ convolutions, so that each location gets its own $F(\boldsymbol\nu_i)$ . Then it classifies each location against the zero-shot prototypes of this section. For a ViT, the class token's query plays the role of the mean query, and MaskCLIP also adds the layer's input back to its output, as the Transformer's residual connection does.

Segmentation quality is measured by mIoU, the overlap between predicted and true masks (intersection over union) averaged over classes (§27.6). With no training and no labels, a CLIP ViT-B/16 read this way reaches 21.7 mIoU on the PASCAL Context benchmark. Reading the ordinary per-location attention outputs the same way gives 9.0. Two training-free refinements, smoothing the keys and removing unlikely class prompts, give 25.5 (Zhou et al. 2022c, Table 1).

Later encoders therefore add losses that read the patch tokens directly. SigLIP 2 adds a decoder that writes captions and predicts boxes from the patch tokens, and another method aligns the mean of the patch tokens with the text, both in §30.4.

Open-vocabulary detection

The simplest working recipe reads almost like a definition (Minderer et al. 2022, a detector called OWL-ViT). Start from a contrastively pretrained pair of encoders, such as CLIP. Remove the image encoder's pooling and final projection. Attach two small heads to every output token, one that predicts a box and one that predicts a class embedding. Score each token's class embedding against the text embeddings of the queries. Then fine-tune the whole model on detection data. The detection heads hold at most 1.1 percent of the parameters, so nearly everything the model knows comes from image-text pretraining.

📐 Open-vocabulary detector (OWL-ViT)

An image is encoded into $N$ output tokens $\mathbf o_1, \dots, \mathbf o_N$ . OWL-ViT has at least 576, more than the 294 objects of the most crowded image in the LVIS dataset. The queries $1, \dots, Q$ are free text, embedded by the text encoder into $\mathbf t_1, \dots, \mathbf t_Q \in \R^{d}$ . Token $n$ predicts

a box $\mathbf b_n \in [0, 1]^4$ from a small MLP, offset so that by default it is centered on the token's own patch,
a class embedding $\mathbf e_n = \mathbf W_{\text{cls}}\mathbf o_n \in \R^{d}$ , from a learned linear map $\mathbf W_{\text{cls}}$ , and
two scalars computed from $\mathbf o_n$ by linear maps, a shift $\psi_n$ and a per-box logit scale $\gamma_n > 0$ . The scale plays the role that $e^{\alpha}$ and the calibrated $\gamma$ played above.

The logit and the score of box $n$ for query $q$ are

\ell_{nq} = \gamma_n\big(\hat{\mathbf e}_n^\top\hat{\mathbf t}_q + \psi_n\big), \qquad p_{nq} = \sigma(\ell_{nq}),

with hats for unit vectors and $\sigma(z) = 1/(1 + e^{-z})$ the logistic sigmoid. So every box-query pair is an independent yes-or-no question.

Training matches predicted boxes to true boxes one to one, by the Hungarian matching of §27.6. The class loss is a focal sigmoid cross-entropy, a version of the cross-entropy that down-weights pairs the model already gets right. At inference the box-query pairs whose scores pass a threshold are kept. The one-to-one matching trains each object to claim one box, so the paper uses no further step. A pipeline may still add non-maximum suppression (NMS) for each query as a safeguard, dropping any kept box that overlaps a higher-scoring box for the same query too much. The exercise below does that.

In plain words, each token proposes one box and says what the box looks like in the language of the text encoder. The score of a query is a cosine, shifted and stretched per box, and nothing is normalized across queries. The per-box shift and scale are in the released implementations, not in the paper's text. At test time OWL-ViT scores each query with the "7 best" CLIP templates and averages the resulting probabilities.

Three design choices deserve their reasons, and interviewers ask about each.

First, the queries never meet the image inside the network. Their embeddings are computed once and cached, so LVIS evaluation scores all 1,203 category names against every image at little cost. Some detectors instead fuse text and image early, so the queries shape the image features, as GLIP (Li et al. 2022) and Grounding DINO (Liu et al. 2023) do. The price is rerunning the fused layers for every image and every query set, the cost trade-off of §30.1.

Second, the scores are sigmoids, not a softmax over the queries. A box can deserve several queries at once ("dog" and "animal"). The query set changes from image to image. And most of the hundreds of boxes cover background and should score low on every query.

Third, the per-box shift is how a box says "I am not an object" whatever the query. A softmax over queries could not use it, since adding the same $\gamma_n\psi_n$ to every logit of a box leaves a softmax unchanged. The same reasoning gives the sigmoid loss of SigLIP its learned bias in §30.4, where matching pairs are rare among all pairs. And because a query is only a vector, an image embedding of an example object can replace the text, which gives one-shot detection of things that are hard to name.

✏️ Softmax over queries lets the background through

Take two boxes and three queries, "dog", "animal", and "car". Box A covers a dog, with cosines $(0.30, 0.28, 0.05)$ . Box B covers grass, with cosines $(0.10, 0.09, 0.08)$ . Take $\gamma_n = 20$ and $\psi_n = -0.2$ for both boxes.

With sigmoids, box A scores $\sigma(2) = 0.881$ for "dog", $\sigma(1.6) = 0.832$ for "animal", and $\sigma(-3) = 0.047$ for "car". Box B scores $\sigma(-2) = 0.119$ , $\sigma(-2.2) = 0.100$ , and $\sigma(-2.4) = 0.083$ . A threshold of 0.3 keeps exactly box A, as a dog and as an animal.

With a softmax over the queries at the same scale, the shift cancels. Box A gets $(0.596, 0.400, 0.004)$ and box B gets $(0.402, 0.329, 0.269)$ . The grass's "dog" score now beats the dog's "animal" score, so no threshold keeps both of the dog's labels and drops the grass. Delete the "car" query and the grass's "dog" score rises to $0.550$ , although nothing in the image changed.

How well does it work on classes it never saw boxes for? LVIS is a detection dataset with 1,203 categories, and its 337 rare ones have only 1 to 10 training images each. OWL-ViT removed every box whose label matches a rare LVIS name from all of its detection training data. So its average precision on the rare classes, $\text{AP}_r$ , is a zero-shot detection score. Average precision (AP) is the area under the precision-recall curve, averaged over classes and over overlap thresholds as in COCO (§27.6).

📌 Open-vocabulary detection on LVIS rare classes

OWL-ViT with a public CLIP ViT-L/14 at 840 px, fine-tuned on the Objects365 and Visual Genome detection datasets, reaches 34.6 AP on all of LVIS and 31.2 $\text{AP}_r$ on the rare classes, the zero-shot part (Minderer et al. 2022, Table 1). Trained instead on LVIS's common and frequent classes alone, the setting of earlier work, the same model gets 25.6 $\text{AP}_r$ . The ensemble version of the earlier ViLD detector, which distills CLIP's embeddings of cropped region proposals into a two-stage ResNet-50 detector (Gu et al. 2021), gets 16.6. Dropping the prompt ensemble at inference cost OWL-ViT 5.5 points of $\text{AP}_r$ in an ablation on a smaller model, so the prompt engineering of this section matters for boxes too.

OWLv2 (Minderer et al. 2023) scaled the detection data by self-training. OWL-ViT L/14 drew pseudo-boxes on about 10 billion WebLI web images, querying each with all the word n-grams of its own caption. A new model trained on those pseudo-boxes and fine-tuned on LVIS's common and frequent classes reached 44.6 $\text{AP}_r$ with a CLIP L/14 backbone, and 47.2 with a SigLIP G/14.

The architecture barely changed between 31.2 and 44.6. The data did, from about two million human-boxed images to billions of machine-boxed ones. That is the same move that made CLIP itself, and the same lesson the robustness evidence below teaches.

Interview

Open-vocabulary detection from per-box embeddings

Write open_vocab_detect(boxes, box_emb, text_emb, scale, bias, thr=0.3, iou_thr=0.5). The inputs are boxes of shape (N, 4) in corner form $(x_1, y_1, x_2, y_2)$ , per-box class embeddings box_emb of shape (N, d), and query embeddings text_emb of shape (Q, d), neither normalized. scale and bias are per-box arrays of shape (N,) or scalars.

Score every box against every query with $p_{nq} = \sigma\big(\gamma_n(\hat{\mathbf e}_n^\top\hat{\mathbf t}_q + \psi_n)\big)$ . Keep the pairs with $p_{nq}$ above thr, and run non-maximum suppression separately for each query. Return three arrays (box_idx, query_idx, score) for the surviving pairs, sorted by descending score.

The setup provides box_iou(A, B) and a class-agnostic nms(boxes, scores, iou_thr) from §27.6. The second returns the kept indices in descending score order.

import numpy as np

def open_vocab_detect(boxes, box_emb, text_emb, scale, bias, thr=0.3, iou_thr=0.5):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Robustness under natural distribution shift

A distribution shift is a change between the images a model was tuned on and the images it meets in use. A natural shift is one that happens without anyone corrupting the images, such as new photographs, sketches, or objects in odd poses. Write ID (in distribution) for the original test set and OOD (out of distribution) for a shifted one.

Here is the most surprising table in the CLIP paper. Take a ResNet-101 trained on ImageNet and zero-shot CLIP ViT-L/14 at 336 px. They have the same ImageNet accuracy, 76.2 percent. Now evaluate both on new test sets of the same 1,000 classes, or subsets of them, collected in other ways.

✏️ Robustness at equal ImageNet accuracy

test set (top-1, %)	ResNet-101	zero-shot CLIP L/14@336px	difference
ImageNet	76.2	76.2	0
ImageNetV2 (a re-collection of ImageNet's test set)	64.3	70.1	+5.8
ImageNet-R (renditions, such as paintings and cartoons)	37.7	88.9	+51.2
ObjectNet (objects in unusual poses and scenes)	32.6	72.3	+39.7
ImageNet Sketch	25.2	60.2	+35.0
ImageNet-A (images that fool a ResNet-50)	2.7	77.1	+74.4

From Figure 13 of Radford et al. (2021). Both models have the same ImageNet accuracy, and ResNet-101 is a standard ImageNet model close to the trend of such models. So the last column is approximately CLIP's effective robustness, defined next.

Reading this table correctly needs a baseline. Taori et al. (2020) evaluated 204 ImageNet models in 213 test conditions and found a regularity. For models trained on ImageNet without special robustness methods, ImageNet accuracy predicts accuracy on a natural shift such as ImageNetV2 almost perfectly. The relation is a straight line after both accuracies are transformed by the logit, $\log\frac{p}{1-p}$ . Almost no intervention escaped that line. Synthetic robustness tricks, such as added noise, style transfer, and adversarial training, did little for natural shifts. The main exception was training on more, and more diverse, data. So "robust" has to mean above the line.

📐 Effective robustness

Let $\operatorname{logit}(p) = \log\frac{p}{1-p}$ . Let $\operatorname{acc}_{\text{ID}}$ and $\operatorname{acc}_{\text{OOD}}$ be a model's accuracies on the original and the shifted test set. Fit $\operatorname{logit}(\operatorname{acc}_{\text{OOD}}) \approx a\,\operatorname{logit}(\operatorname{acc}_{\text{ID}}) + c$ by least squares on a baseline family of models trained on the in-distribution data, with scalars $a$ and $c$ . The effective robustness of a model is

\operatorname{ER} = \operatorname{acc}_{\text{OOD}} - \operatorname{logit}^{-1}\big(a\,\operatorname{logit}(\operatorname{acc}_{\text{ID}}) + c\big), \qquad \operatorname{logit}^{-1}(z) = \frac{1}{1 + e^{-z}} .

Relative robustness is the plain change in $\operatorname{acc}_{\text{OOD}}$ caused by an intervention (Taori et al. 2020).

In plain words, effective robustness is how much better a model does under the shift than standard models with the same in-distribution accuracy. For example, standard models at 76.2 percent ImageNet accuracy reach about 37.7 percent on ImageNet-R, as ResNet-101 does. Zero-shot CLIP reaches 88.9 percent at the same ImageNet accuracy, an ER of about 51 points. A model on the line has ER 0, however accurate it is. A useful intervention should raise both effective and relative robustness, since a method can climb above the line just by losing in-distribution accuracy.

Why logits? The transform stretches accuracies near 0 and 1, where a linear axis squeezes every model into a corner, and Taori et al. (2020) found it gave the best linear fits. That is an empirical finding, but a simple model shows why some such transform should straighten the trend.

∂ Why ID and OOD accuracies line up after a probit or logit transform

Suppose each model's margin on a random test example (the correct logit minus the largest wrong one) is approximately normal, with mean $\mu$ and standard deviation $\sigma$ in distribution. Then $\operatorname{acc}_{\text{ID}} = \Pr(\text{margin} > 0) = \Phi(\mu/\sigma)$ , where $\Phi$ is the standard normal CDF. Suppose the shift acts the same way on every model of the family. It scales and lowers the margin to mean $\kappa\mu - \delta$ and standard deviation $\sigma'$ , with $\kappa\sigma/\sigma'$ and $\delta/\sigma'$ the same for all models. Then

\Phi^{-1}(\operatorname{acc}_{\text{OOD}}) = \frac{\kappa\mu - \delta}{\sigma'} = \frac{\kappa\sigma}{\sigma'}\cdot\frac{\mu}{\sigma} - \frac{\delta}{\sigma'} = \frac{\kappa\sigma}{\sigma'}\,\Phi^{-1}(\operatorname{acc}_{\text{ID}}) - \frac{\delta}{\sigma'},

a straight line in probit coordinates, the coordinates given by $\Phi^{-1}$ . The logistic CDF is close to a rescaled normal CDF, $\operatorname{logit}^{-1}(1.702z) \approx \Phi(z)$ , so the line is nearly straight in logit coordinates too. A model above the line is one that the shift hurts less than the family's shared $\kappa$ , $\delta$ , and $\sigma'$ say it should. This is a model of the regularity, not a theorem about real networks.

On these axes every zero-shot CLIP model sits far above the ImageNet line. Radford et al. (2021) report that zero-shot CLIP shrinks the gap between ImageNet accuracy and accuracy under the shifts by up to 75 percent.

Then comes the twist. Adapting CLIP to ImageNet with an L2-regularized logistic-regression probe on its frozen features raises ImageNet accuracy by 9.2 points, to 85.4 percent. Yet it slightly lowers the average accuracy across the shifts. ImageNetV2, collected the same way as ImageNet, gains 5.8 points, while ImageNet-R loses 4.7, ObjectNet 3.8, ImageNet Sketch 2.8, and ImageNet-A 1.9. The probe learned what is specific to ImageNet's photographs. The same paper found that few-shot probes on CLIP's features keep much of the effective robustness, and that it fades as more ImageNet labels are used.

Core

Effective robustness from a model zoo

Write effective_robustness(id_base, ood_base, id_new, ood_new, eps=1e-3). The first two arguments are the in-distribution and shifted accuracies, as fractions in $[0, 1]$ , of a baseline family of models. The last two are the same accuracies for the models to evaluate.

Before every logit, clip the accuracy to $[\epsilon, 1 - \epsilon]$ . Fit $\operatorname{logit}(\text{OOD})$ against $\operatorname{logit}(\text{ID})$ by least squares on the baseline. Return (er, a, c), the effective robustness of each new model in accuracy units, and the fitted slope and intercept. The effective robustness subtracts the prediction from the observed OOD accuracy as given, unclipped.

import numpy as np

def effective_robustness(id_base, ood_base, id_new, ood_new, eps=1e-3):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

What causes the robustness

CLIP differs from an ImageNet ResNet in five ways at once. It saw 400 million images instead of 1.28 million. Its images came from a different and far more varied distribution. It was supervised with language during training. It uses language at test time, through prompts. And it was trained with a contrastive loss. Any of these could be the cause.

Fang et al. (2022) tested each one in controlled experiments. Their key tool was ImageNet-Captions, 463,622 ImageNet training images paired with their original Flickr titles, descriptions, and tags. It lets you train with captions or with labels on exactly the same images.

In plain words, language matters because it is the cheap way to supervise hundreds of millions of varied web images, not because words make features robust. This puts the burden on the data, which is why §30.5 treats dataset curation as a science. That section also finds that zero-shot accuracy on a concept tracks how often the concept appears in pretraining. A concept CLIP rarely saw is one it will classify poorly, robust or not.

Fine-tuning without losing it

Most applications want more than zero-shot accuracy. So they fine-tune on labeled target data, and the robustness tends to disappear. Wortsman et al. (2022) measured this for CLIP ViT-L/14@336px. Zero-shot, it scores 76.2 on ImageNet and 73.3 on average over five shifts (ImageNetV2, ImageNet-R, ImageNet Sketch, ObjectNet, ImageNet-A). Fine-tuned end to end on ImageNet, it reaches 86.2 on ImageNet but falls to 68.6 on the shifts, below where it started.

Their fix costs nothing at training or at inference. Average the weights of the zero-shot model and the fine-tuned model.

📐 WiSE-FT, the weight-space ensemble

Let $\boldsymbol\theta_0$ be the zero-shot model's parameters, the image encoder together with the zero-shot head $e^{\alpha}\mathbf W$ built from the class prompts. So $\boldsymbol\theta_0$ is a complete classifier with nothing trained. Fine-tune it on labeled target data, starting at $\boldsymbol\theta_0$ , to get $\boldsymbol\theta_1$ , the same architecture with every entry possibly changed. For a mixing coefficient $\lambda \in [0, 1]$ , the WiSE-FT model has parameters

\boldsymbol\theta_\lambda = (1 - \lambda)\,\boldsymbol\theta_0 + \lambda\,\boldsymbol\theta_1 ,

so $\lambda = 0$ is the zero-shot model and $\lambda = 1$ is the fine-tuned one. Wortsman et al. (2022) recommend $\lambda = 0.5$ when nothing is known about the target shift. On ImageNet and its five shifts, the best fixed $\lambda$ chosen separately for each test set was only 0 to 0.4 points better than $0.5$ . The end-to-end variant fine-tunes every parameter, and the linear-classifier variant fine-tunes only the head on frozen features.

In plain words, average the two models entry by entry. For a nonlinear network there is no general reason for an average of two weight vectors to be a good network, and averaging two independently trained networks typically fails. Here it works because the fine-tuned model started at $\boldsymbol\theta_0$ and stayed in the same valley of the loss, so accuracy stays high along the straight path between them. That is an empirical finding of the paper. For a linear head, the interpolation is exact in another sense.

Proof. A matrix times a vector is linear in the matrix, so the product distributes over the sum and commutes with the scalars. $\square$

✏️ WiSE-FT at real scale

Here are the numbers for CLIP ViT-L/14@336px on ImageNet, with the shift column averaged over the five shifts (Wortsman et al. 2022, Table 1).

model	ImageNet	average of 5 shifts
zero-shot	76.2	73.3
linear probe on ImageNet (the CLIP paper's)	85.4	71.8
fine-tuned end to end	86.2	68.6
WiSE-FT, end to end, $\lambda = 0.5$	86.8	76.9

The interpolated model beats the fine-tuned one in both columns, and beats the zero-shot one by 3.6 points under shift. The authors also found that small changes in fine-tuning hyperparameters, such as the learning rate, move OOD accuracy by several points while ImageNet accuracy barely changes. So robustness cannot be tuned by watching the in-distribution validation set.

The widget puts these published points on the logit axes of effective robustness. Pick a shift and tilt the baseline line. The dashed path joins the zero-shot model, WiSE-FT at $\lambda = 0.5$ , and the fine-tuned model.

A second method changes the fine-tuning loss instead (Goyal et al. 2023, a method called FLYP). Write each class as its prompt. Then keep training both encoders with the contrastive loss used in pretraining, with the class prompts as the texts, rather than attaching a cross-entropy head. On CLIP ViT-B/16, across seven OOD datasets (two from the WILDS benchmark and five ImageNet shifts), it gained 4.2 points OOD over standard fine-tuning on average. It also beat the previous best method by over 1 point both in and out of distribution. That method, LP-FT, fits a linear probe first and then fine-tunes everything (§28.6). The lesson both papers share is to stay close to the pretrained model, in weights or in objective.

In PyTorch

The reference block builds a zero-shot head as an nn.Linear without bias. It calibrates the head's scale with L-BFGS, a quasi-Newton optimizer, on the log of the scale, which keeps the scale positive. Then it fine-tunes a copy and checks the linear-head identity of WiSE-FT numerically.

🔥 In PyTorchfor a machine with PyTorch

The calibrated scale is about 8, a twelfth of CLIP's 100, and it cuts the NLL from 4.07 to 0.62 without changing a prediction. The interpolated head reproduces the averaged logits to within $10^{-13}$ . This toy has no distribution shift, so it says nothing about robustness, only about the mechanics.

With a real checkpoint, the whole zero-shot pipeline is a few lines of OpenCLIP, an open-source library of CLIP models. This block is shown, not run, since it downloads weights.

Is it really zero-shot

Two different worries hide in the word. The first is contamination, test images sitting among the 400 million training pairs. CLIP's authors ran a duplicate detector over 35 evaluation datasets, and §22.3 covers such audits in general.

So contamination moved CLIP's numbers by fractions of a point. The box below has the largest cases and the audit's main weakness.

The second worry is subtler and does not show up as duplicates. "Zero-shot" in CLIP's sense means the dataset's training split was not used, not that the concepts were unseen. Every ImageNet class appears many times in web captions, and accuracy on a concept rises with how often it was seen, which §30.5 quantifies. Add the tuning of templates and class names against ImageNet's training labels described earlier. The honest description of CLIP's 76.2 is no gradient steps on ImageNet, a little human tuning against its labels, and broad prior exposure to its concepts. In an interview, say "no task-specific training", never "never seen".

⚠️ Where zero-shot evaluations go wrong

Ranking captions without normalizing lets long text vectors win every row. Treating only the first of five captions as the positive turns image-to-text R@1 into a much harder task than the one papers report. Breaking ties with argsort silently favors the model, and a broken model that embeds everything to one point would score perfectly under a favorable tie rule. Reading softmax probabilities at scale 100 as confidences gives near-certain answers that are wrong far more often than they claim. Reporting a 1-point gain on Flickr30k's 1,000 images as progress ignores a 2-point interval. Tuning prompts on the test set makes the number neither zero-shot nor reproducible. And fine-tuning a robust model with a learning rate chosen on in-distribution validation accuracy can lose many OOD points with no sign on the validation curve.

🧭 What to remember

A zero-shot classifier is a linear head on frozen image features whose class weights are normalized, ensembled text embeddings, $\mathbf z = e^{\alpha}\mathbf W\hat{\mathbf u}$ . At the PMI optimum it assumes a uniform class prior, and adding $\log\pi_k$ corrects for a known one.
The logit scale of 100 was learned for caption matching. Calibrate it with one temperature fitted on held-out NLL, which leaves the predictions and the accuracy unchanged.
Prompts are weights. Templates, context, and unambiguous class names add points. Ensembling in embedding space is a rescaled logit average that costs nothing at inference (80 templates, 3.5 points on ImageNet). Descriptions written by a language model help further, and learned prompts help most in distribution but overfit the classes they saw.
Zero-shot is worth about four labeled shots per class on average and loses to fully supervised probes by 10 to 25 points. The best few-shot classifiers use the text as a prior whose weight shrinks as labels arrive.
Image-text recall at K is a hit rate with five captions per image, reported in both directions, and its sampling error on Flickr30k is about 2 points. A few hub items win many queries, and the inverted softmax over a bank of queries removes each item's shared score, worth 4.5 points of COCO text-to-image R@1 for CLIP ViT-B/32.
CLIP's patch tokens are poorly localized because its loss reads only the pooled vector. Open-vocabulary detectors put a box head and a class-embedding head on every token, score boxes against text with independent sigmoids and a per-box bias, and gain most from more detection data, as self-training on web-scale pseudo-boxes showed (LVIS rare-class AP from 31.2 to 44.6).
Effective robustness is accuracy above the logit-linear trend of standard models. CLIP's comes mainly from the diversity of its training data, ImageNet probes and fine-tuning erode it, and WiSE-FT and FLYP keep most of it. Next, §30.4 replaces the batch softmax with a sigmoid loss, and §30.5 asks what data makes these classifiers good.

Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. Section 3 covers prompts, ensembling, few-shot comparisons, and robustness, Section 5 the overlap audit, and Appendix E.1 the retrieval numbers.
Taori et al. (2020). Measuring Robustness to Natural Distribution Shifts in Image Classification. The 204-model testbed, the logit-linear trend, and the definitions of effective and relative robustness.
Fang et al. (2022). Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP). The controlled experiments that isolate the training distribution as the cause.
Wortsman et al. (2022). Robust fine-tuning of zero-shot models. WiSE-FT, with careful measurements of how fine-tuning hyperparameters trade in-distribution and OOD accuracy.
Goyal et al. (2023). Finetune like you pretrain: Improved finetuning of zero-shot vision models. FLYP, fine-tuning with the pretraining loss and class prompts.
Zhou et al. (2022a). Learning to Prompt for Vision-Language Models. CoOp, learned context vectors for the text encoder.
LeVine et al. (2023). Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models. Measures CLIP's zero-shot miscalibration and shows that one temperature per model transfers across datasets.
Minderer et al. (2022). Simple Open-Vocabulary Object Detection with Vision Transformers. OWL-ViT, a CLIP encoder turned into a detector by putting box and class heads on every token, with the LVIS rare-class protocol and careful ablations.
Minderer, Gritsenko, and Houlsby (2023). Scaling Open-Vocabulary Object Detection. OWLv2 and OWL-ST, self-training on billions of pseudo-boxed web images.
Zhou, Loy, and Dai (2022c). Extract Free Dense Labels from CLIP. MaskCLIP, why CLIP's dense outputs fail and how its value embeddings segment without training.
Bogolin et al. (2022). Cross Modal Retrieval with Querybank Normalisation. Hubness in cross-modal retrieval, the inverted softmax and its relatives, and the dynamic inverted softmax.

Made for larger screens

Zero-shot classification and retrieval

A zero-shot classifier is a linear head

The class prior the head assumes

Prompts are part of the classifier

Averaging many prompts in embedding space

Calibrating the logit scale

Calibrate a zero-shot classifier

Thousands of classes, in chunks

Top-1 and top-5 accuracy over a large label set, in chunks

How many labeled images is a class name worth

Combining the text with the shots

A probe that starts from the text

Image-text retrieval, measured correctly

Recall at K for image-text retrieval with five captions per image

Hubs, and normalizing over the queries

Find the hubs and normalize them away

From images to regions

Why CLIP's patch tokens are poorly localized

Open-vocabulary detection

Open-vocabulary detection from per-box embeddings

Robustness under natural distribution shift

Effective robustness from a model zoo

What causes the robustness

Fine-tuning without losing it

In PyTorch

Is it really zero-shot

Read more

Watch next