This section is about what CLIP gets wrong, and why. CLIP is a pair of encoders, one for images and one for text, trained on web captions so that a photo and its caption get vectors that point the same way (§30.1). It is excellent at matching photos with captions. It is poor at word order, counting, and spatial relations, and it has other blind spots that a person would never have. Each failure has a cause we can state exactly, and the cause tells you which fix will work. This matters beyond CLIP, because most vision-language models see images through a CLIP-style encoder and inherit its blind spots.

Two small tests show the problem. Show CLIP a photo of a mug standing on a book, and ask which caption fits, "a mug on a book" or "a book on a mug". Show it a photo of three dogs, and ask whether it shows three dogs or six. CLIP does poorly on both. Yet it is superb at ordinary matching. COCO is a standard set of photos with five human-written captions each. Given a COCO test photo, CLIP's best model, ViT-L/14 at 336 pixels, ranks the right caption first among all 25,000 test captions 58.4 percent of the time. It also names ImageNet classes as accurately as a ResNet-50 trained on ImageNet's own labels (Radford et al. 2021).

The failures are large. Winoground is a benchmark of 400 examples, each with two photos and two captions that use the same words in a different order. Take CLIP ViT-B/32, the smallest of CLIP's Vision Transformer models, which cuts each image into patches of 32 by 32 pixels (§27.1). It picks the right photo for both captions of an example 10.5 percent of the time, while random guessing gives 25 percent (Thrush et al. 2022). Asked how many objects a photo shows, with nine possible answers, it is right 32 percent of the time (Paiss et al. 2023).

The cause is not too little training. Most of the failures follow from the training objective (§30.1 and §30.2) and from the web data (§30.5). We take them in order. First comes geometry, the modality gap that keeps images and texts in separate regions of the sphere. Then word order, with an exact account of what Winoground measures. Then counting and spatial relations, social bias in zero-shot labels, attacks that write words on objects, the short text window, and photos that hold several objects. Each part ends with the fixes the field tried and what the evidence says about them.

The training loss, in brief. A batch holds $B$ photos and their $B$ captions. The score of a photo and a caption is the cosine of their embeddings times a learned number, the logit scale. The row loss asks each photo to pick out its own caption among the $B$ captions, with a softmax over its row of scores. The column loss asks each caption to pick out its own photo the same way. The CLIP loss is the average of the two, the two-way form of the InfoNCE loss of §7.2 (§30.2 trains with it at scale). Almost every limit below traces back to what this loss does and does not ask for.

Notation. $x$ is an image and $y$ a text. $f$ and $g$ are the image and text encoders, often called the two towers. $\mathbf u_i = f(x_i)$ and $\mathbf v_j = g(y_j)$ are embeddings in the joint space $\R^d$, and a hat marks a vector scaled to length one, as in $\hat{\mathbf u}_i$. The logit scale is $e^{\alpha} = 1/\tau$, where $\tau$ is the temperature, so the score of image $i$ and text $j$ is $s_{ij} = e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j$. The rows of $\hat{\mathbf U}, \hat{\mathbf V} \in \R^{B\times d}$ are the batch's unit image and text embeddings, and $\mathbf S = e^{\alpha}\hat{\mathbf U}\hat{\mathbf V}^\top$ is the $B\times B$ matrix of scores, with images as rows and captions as columns. No attention appears in this section, so $\hat{\mathbf V}$ is never the value matrix.

Images and texts sit in two separate clouds

Embed a few thousand COCO photos and their captions with CLIP, then project the vectors to two dimensions to look at them. You might expect one cloud, with each photo next to its caption. You see two clouds instead, one of images and one of texts, with empty space between them. Each matching pair reaches across that space. Liang et al. (2022) named this the modality gap. They found it consistently across the multimodal contrastive models they examined, which covered text, natural images, videos, medical images, and amino-acid sequences.

The gap is measured with centroids. The centroid of a set of vectors is their average, the center of the cloud.

In plain words, the gap is large when images are much more similar to other images, and texts to other texts, than images are to texts. Each modality crowds into its own narrow region of the sphere.

For CLIP on COCO, Liang et al. measured a gap of $\lVert\Delta_{\text{gap}}\rVert = 0.82$. That number explains something that puzzles people the first time they use CLIP. The cosine between a photo and a correct caption is small, far below 1. LAION, the open web dataset of §30.5, keeps every pair whose CLIP cosine reaches 0.3 (0.28 for English in LAION-5B), a threshold its builders chose by inspecting examples. A matched pair is close compared with the other captions, yet far in absolute terms, because every caption sits on the far side of the gap.

The first two softmaxes agree to six digits, and the third does not. Adding one vector to every caption changes nothing that a row softmax can see. Only putting the shifted captions back on the circle changes the scores. This pattern is the core of the theory of the gap, and the next three parts build on it.

The gap starts at initialization

Liang et al. traced the gap to two causes. The first is present before any training. A randomly initialized deep network maps all its inputs into a narrow cone, a small region of the sphere in which every pair of outputs has a large positive cosine. A ReLU is enough to cause it. After a ReLU every coordinate is zero or positive, so any two hidden vectors have a dot product of zero or more. Stacking more such layers tends to pull their directions closer still. Two towers initialized independently each get their own cone, pointing in unrelated directions. So image and text embeddings start far apart.

Trained word vectors crowd into a cone too, an effect called anisotropy (§1.9). There it is usually blamed on skewed word frequencies. Here neither trained weights nor skewed data are needed. The cell builds two random towers of ReLU layers, feeds both the same inputs, and measures the crowding within each tower and the gap between them.

With no hidden layer the outputs are spread over the sphere. The mean cosine within a tower is 0.001, and the two towers' centroids nearly coincide (a gap of 0.08). With eight ReLU layers the mean cosine within a tower is 0.74, and the gap between the towers is 1.26, before a single gradient step.

Liang et al. found the same at full scale. Two outputs of a randomly initialized ResNet have an average cosine of 0.99. Even the trained CLIP towers keep average pairwise cosines of 0.47 for images and 0.51 for texts on COCO. When they trained CLIP from scratch on COCO, the gap was 1.19 at initialization and 1.30 after training. In a second run they first rotated the text embeddings onto the image embeddings, so the gap at initialization was only 0.04. Training reopened it to 0.75. Initialization explains where the gap starts. Something in the loss keeps it open.

The loss cannot see where each cloud sits

The second cause is a symmetry of the loss. Recall the example. Adding one vector to every caption adds a constant to each row of scores, and a softmax ignores a constant added to all its inputs. So the row loss cannot see where the text cloud sits as a whole. By the same argument, the column loss cannot see where the image cloud sits.

The theorem below makes this exact. It uses one new quantity. The load of image $i$ is the total probability it collects from the column softmaxes, summed over all $B$ captions. Each column softmax hands out one unit of probability, so the loads add up to $B$, and in a perfectly balanced batch every image collects exactly 1.

In plain words, the only force on the text cloud as a whole comes from the column loss. That force rebalances the images. It pulls captions toward images that are retrieved too little and away from images that collect too much. It never points from the text centroid toward the image centroid. Nothing in the loss asks the two clouds to meet.

The theorem is exact for raw dot products. CLIP normalizes both embeddings, and a shift followed by renormalization changes the shifted vectors by different amounts, so for CLIP the invariance is approximate (the third softmax in the first cell). It still describes the main effect. The next cell checks both parts of the theorem with random embeddings, including the claim that moving the whole image cloud leaves the gradient unchanged.

Each half prints the same loss before and after the offset, the formula matches finite differences, and moving every image by a far vector leaves the gradient as it was. For practice this means more training will not close the gap by itself, since no term of the loss pulls on it. An interviewer who asks why the gap survives training wants this argument.

With noisy pairs the loss prefers a gap

Blindness explains why nothing closes the gap. Liang et al. went further. They showed that a real CLIP, at its learned temperature, has its lowest loss with the gap open. They shifted every image by $-\lambda\Delta_{\text{gap}}$ and every text by $+\lambda\Delta_{\text{gap}}$, renormalized, and measured the loss on COCO for a range of $\lambda$.

At $\tau = 1/100$, the temperature CLIP learns, the original gap gave the lowest loss, and closing it made the loss worse. At higher temperatures the best gap moved toward zero. Fine-tuning CLIP on COCO agreed. At $\tau = 1/10$ or $\tau = 1$ fine-tuning shrank or closed the gap, and at $\tau = 1/100$ it did not. A small simulation of theirs traced the effect to mismatched pairs, where an image sits closer to someone else's caption than to its own. Web data is full of them.

Why would the loss want a softer softmax? CLIP caps its logit scale at 100 during training (§30.2), and the learned scale reaches that cap. A sharp softmax pays heavily for every confident mistake, and mismatched pairs force such mistakes. A model stuck at scale 100 can still soften its predictions by opening a gap. The cell searches over the latitude of the captions for the lowest loss, with clean pairs and with two captions swapped, at scales 100 and 10.

With clean pairs the loss is lowest with the gap closed, at either scale. With two of the six captions swapped and the scale at 100, the best latitude is 70 degrees, a gap of 1.11. Its loss, 0.811, is exactly the loss of closing the gap and setting the scale to 34.2 by hand. At scale 10 the softmax is already soft, and the gap closes again. This is Liang et al.'s temperature effect, reproduced in three dimensions. In practice it means the gap is not a bug that more training removes. It is how a temperature-capped loss copes with noisy pairs.

The widget scales this up to 24 pairs in four classes. Dots are images on the equator, and squares are their captions. A thin line joins each image to its own caption, so the swapped pairs cross. The slider lifts every caption to the same latitude. The curve on the right is the CLIP loss at every latitude, with its minimum dashed. The numbers below score three uses of the embeddings, ranking captions for each image (R@1), zero-shot classification, and search over a mixed pool of images and captions.

Before you move the slider, predict where the loss will be lowest with some captions swapped and the scale at 100, and whether zero-shot accuracy will change as you move it.

Then switch to clean pairs and check which of the three numbers under the picture move at all. Zero-shot accuracy should not, for a reason the next part makes exact.

What the gap breaks and what it leaves alone

Some uses are immune to the gap. Zero-shot classification compares one image's scores against $K$ class prompts. Adding the same vector $\mathbf c$ to every prompt adds $e^{\alpha}\hat{\mathbf u}^\top\mathbf c$ to every class's logit, so the argmax and the softmax do not change. This is exact for raw dot products and approximate after renormalization. The same holds for any retrieval that ranks one modality against a query from the other, which is what the retrieval benchmarks measure. In the widget, zero-shot accuracy and image-to-text recall never move as the gap opens.

Shifting both clouds and renormalizing is different, because renormalization changes each class's logit by a different amount. Liang et al. tried shifts of several sizes in both directions on five zero-shot tasks. The best shift raised accuracy slightly on each, by widening the gap on some datasets and narrowing it on others. The largest gain was on the satellite images of EuroSAT, from 54.10 to 56.45 percent.

They also ran CLIP's fairness probe, described later in this section, which counts face photos given a denigrating label. Widening the gap from 0.82 to 0.97 lowered that count for every race group. Top-1 accuracy on the faces moved only from 58.17 to 57.39 percent. So the gap is a setting with effects, not only a defect to remove.

The clearest damage is to anything that compares across modalities in absolute terms. Three uses suffer.

• Search over a mixed pool. A search over a pool of both images and captions returns captions first. Across a gap, a caption is usually closer to other captions than to any image. In the widget, mixed-pool recall falls to 0 once the captions are lifted by 12 degrees, a gap of only 0.21.
• One threshold for all pairs. A single similarity threshold means different things for image-text pairs and for text-text pairs.
• Text in place of an image. A text embedding is not a usable stand-in for an image embedding. DALL-E 2 (Ramesh et al. 2022) draws an image from a CLIP image embedding. Rather than feed its decoder the caption's text embedding, it trains a prior, a model that maps a caption to a CLIP image embedding. The authors report that feeding the text embedding directly gives reasonable images, and that the prior gives better ones. Text-to-image generators return to this question.

import numpy as np

def modality_gap(U, V):
    raise NotImplementedError

def shift_embeddings(U, V, lam):
    raise NotImplementedError

Word order, or who did what to whom

A bag-of-words encoder ignores word order. The simplest one averages the vectors of the words in a caption, the order-free pooling that §1.9 listed among the weaknesses of word vectors. Under such an encoder, "a mug on a book" and "a book on a mug" contain the same five words, get the same vector, and receive the same score against every image. No photo, however clear, could make the model prefer one of them.

CLIP's text tower is not a bag of words. It is a Transformer with position embeddings and a causal mask, and it reads its output at the end-of-text token. So it can represent order. The question is whether training ever asks it to.

Yuksekgonul et al. (2023) answered with an experiment. They took the COCO and Flickr30k retrieval benchmarks and destroyed the order information, by shuffling all the words of every caption or the patches of every image. Retrieval accuracy barely fell for most models. So a model that ignores order can do well on these benchmarks, and contrastive pretraining is the same task at a larger scale. In a batch of 32,768 web captions, the wrong captions almost never share all their words with the right one. An encoder that treats each caption as a bag of words gets almost the same loss as one that reads order. Gradient descent has little reason to build a feature that the loss almost never rewards, so order is barely learned.

This changes how you read benchmark results. A high retrieval score says little about whether a model understands who did what to whom. An interviewer who asks whether CLIP understands language is looking for this argument about the objective, not one about the architecture.

Winoground measures order with three scores

Winoground (Thrush et al. 2022) makes order the only thing that matters. Each of its 400 examples has two images and two captions with exactly the same words in a different order, such as "some plants surrounding a lightbulb" and "a lightbulb surrounding some plants". Each caption matches one of the images. Expert annotators wrote and checked every example. The benchmark has three scores, one for each direction of choice and one for both.

The bag-of-words failure is now a short theorem.

Proof. The two captions of an example are reorderings of each other, so $g(C_0) = g(C_1)$, and the two rows of $\mathbf S$ are equal, $S_{0i} = S_{1i}$ for both images. The text test needs $S_{00} > S_{10} = S_{00}$, which is false. With equal rows, the image test needs $S_{00} > S_{01}$ and $S_{01} > S_{00}$, a contradiction. The group test needs both. $\square$

CLIP is not exactly order-blind, so it does not score exactly zero. Its scores are in the fact box below, and they have a curious pattern. The text score is a little above chance, and the image score is well below chance. A model that knew nothing would get 25 percent on both. How can a model do worse than nothing at choosing between two photos? Splitting the four scores into parts answers it.

The decomposition is easiest to trust on four real numbers. Here it is on one example scored by a model that is nearly blind to word order.

In plain words, the text test asks the matching signal to beat the difference between the two captions, and the image test asks it to beat the difference between the two photos. A nearly order-blind encoder makes the two captions almost equal, so $\omega$ and $\eta_0$ are both tiny, and which one wins is close to a coin flip. That puts the text score near chance. The two photos, though, differ in background, lighting, and framing, so $\lvert\xi_0\rvert$ is much larger than the tiny $\omega$, and the image score falls toward zero. This is CLIP's pattern.

The next cell builds a synthetic Winoground with a knob for how much order the text encoder sees. In each example, both captions and both photos share one vector for the common words. The two photos add a weak cue for who is on top, with opposite signs, plus their own random detail. The captions add the same cue scaled by the knob, together with a larger direction that has nothing to do with the photos.

At order 0 the encoder is a bag of words, and all three scores are exactly 0, as the theorem says. With a faint order signal (0.03 to 0.1) the toy lands near CLIP's real numbers, a text score of 0.31 to 0.32 and an image score of 0.05 to 0.15. The printed effects show why. The image effect $\lvert\xi_0\rvert$ stays near 0.03 at every setting, because it comes from the photos. The interaction $\omega$ grows with the order knob but stays below the image effect on average, even at the strongest setting, where the image test passes a third of the time.

Notice also that the text score hardly moves as the order signal grows. Its test compares two small quantities that grow together. So the text score is the wrong number to watch when you try to fix order sensitivity, and the image and group scores are the right ones. An interviewer who asks how you would measure progress on compositionality wants this point.

The widget is the same toy with 400 examples in 32 dimensions. The table shows the four scores of one example, and the bars show the three parts of its decomposition, with a pass or fail for each test. The buttons choose the text encoder, and the slider sets how clearly the photos show the relation. At the default setting the faint encoder scores 33.0, 11.5, and 7.25 percent, close to CLIP's numbers in the box below.

With only 400 examples, how sure are we of these numbers? The Wilson interval of §0.11 is a confidence interval for a proportion that stays accurate for small samples and for rates near 0 or 1. It gives the answer in one line per score.

The text score's interval, 26.4 to 35.4 percent, barely excludes chance. The image score's interval, 7.9 to 13.9, lies entirely below chance. Thrush et al. report wider intervals, 25.0 to 36.5 for the text score. They split the 400 examples into four groups of 100 and put a t-distribution on the four group scores, a different method with only three degrees of freedom. Either way the conclusion holds. So a claimed improvement of 3 points on a benchmark this size is inside the noise. The exception is a comparison of two models on the same examples with a paired test, which §0.11 recommends.

import numpy as np

def winoground_scores(S):
    raise NotImplementedError

What does it take for a text encoder to see order at all? Position information has to interact with the words. Adding a position vector to each word vector and then averaging is still a bag of words. The average splits as $\frac1L\sum_p(\mathbf e_{t_p} + \mathbf p_p) = \frac1L\sum_p\mathbf e_{t_p} + \frac1L\sum_p\mathbf p_p$, where $\mathbf e_{t_p}$ is the vector of the word at position $p$ and $\mathbf p_p$ the vector of position $p$. The second sum is the same whichever word sits where. A Transformer escapes this, because attention mixes words and positions nonlinearly before pooling (§5.4). The next exercise uses the simplest escape, an elementwise product of position and word.

import numpy as np

def bow_encode(tokens, E):
    raise NotImplementedError

def order_encode(tokens, E, P):
    raise NotImplementedError

Hard negatives that differ only in order

Winoground is small and hard. Yuksekgonul et al. built a larger probe, ARO (Attribution, Relation, and Order), with more than 50,000 test cases. They come from COCO, Flickr30k, and Visual Genome, a set of photos annotated with their objects, attributes, and relations. The Visual Genome Relation task asks a model to choose between "the horse is eating the grass" and "the grass is eating the horse" for a crop of the scene. Visual Genome Attribution asks it to choose between "the crouched cat and the open door" and "the open cat and the crouched door". The two Order tasks, on COCO and Flickr30k, ask it to choose between a caption and four shuffled versions of it.

The diagnosis suggests the fix. If the loss never meets two captions with the same words in a different order, give it some. A hard negative is a wrong caption chosen to be close to the right one, so the model must notice the difference to rank it lower. Yuksekgonul et al.'s method, NegCLIP, fine-tunes CLIP on COCO with two kinds of hard negative. For every caption it prepares up to five negatives, each made by swapping two of its noun phrases, nouns, adjectives, adverbs, or verb phrases, and samples one of them for each batch. For every image it adds a strong alternative image, one of the image's three nearest neighbors by CLIP image similarity. The swapped captions join the row softmax only, since no image should be matched to them.

The next two cells reproduce the effect in a toy world where we can see everything. Twenty objects make 380 scenes of the form "a on b". A photo is the sum of the two objects' looks, plus a weak cue for which one is on top, plus noise. A caption is a bag of words plus two slots that record which noun came first and which came second. So the text tower can learn order if anything asks it to. We train the same linear dual encoder twice, with and without the swapped caption of every scene as an extra column. Then we run every unordered pair of objects as a Winoground example.

The gradient on the logits is the one from §30.2, with the row softmax taken over all $2B$ columns. It flows back through the normalization, whose Jacobian removes the component along each vector, and through the projections, exactly as in the training cell of §30.1.

Without hard negatives the model learns some order, since the position slots are free to use, and reaches a group score of 0.56. With the swapped captions in every batch it reaches 0.78, and every score rises. The reason is how rarely ordinary training sees an order-only contrast. In a batch of 16 drawn from 380 scenes, the swapped twin of a given scene is present only about 4 percent of the time (15 chances in 379). The hard negatives put that contrast in every row. At CLIP's real scale a web caption's word-swapped twin is almost never in the batch at all, so ordinary training supplies almost none of this signal.

import numpy as np

def swap_negative(tokens, tags):
    raise NotImplementedError

def clip_loss_with_negatives(U, V, Vneg, scale):
    raise NotImplementedError

Some benchmarks can be passed without looking

The story has a twist. Hsieh et al. (2023) looked at the negatives in ARO and similar benchmarks. Many were ungrammatical or implausible, such as "the grass is eating the horse", or word salad from shuffling. A blind model never sees the image. It picks whichever caption a text-only model judges more plausible or more grammatical. Such blind models beat the best of 17 CLIP models on 9 of 10 tasks in these benchmarks. So training with negatives made by fixed rules can teach the text tower to spot the rules' artifacts rather than to compose meaning.

Their benchmark, SugarCrepe, has 7,512 examples. A language model (ChatGPT) rewrites each true caption into a negative and is told to keep it fluent and plausible. People then remove negatives that are actually true. Finally the set is filtered adversarially, until plausibility and grammar scores no longer separate the right captions from the wrong ones.

On SugarCrepe the gains shrink. On its swap negatives, CLIP ViT-B/32 scores 63.27 percent, the COCO fine-tuned model 70.83, the released NegCLIP 75.33, and humans 99.50. Hsieh et al. also fine-tuned three more NegCLIP models, each with a different kind of rule-made negative. Across these four models and four kinds of negative in ARO and a similar benchmark, CREPE, NegCLIP beat plain fine-tuning by more than 10 points in 11 of the 16 cases. On SugarCrepe it never gained more than 10.

The cell shows the blind trick. A bigram language model scores a sentence by how often each word follows the previous one in its training text. This one is trained only on captions and never sees an image.

Without looking at a single image, the bigram model rejects shuffled captions 97 percent of the time and implausible swaps every time. It is near chance (0.54) only when the swap is as plausible as the original. Only that last kind of negative measures composition. The lesson holds for every multimodal benchmark. Before you believe a vision-language score, run a text-only baseline and, where it makes sense, an image-only one. Interviewers ask exactly this when you present a benchmark result.

Counting, spatial relations, and images CLIP cannot tell apart

CLIP counts poorly

CLIP's authors listed counting among its weak tasks. Paiss et al. measured it with CountBench, 540 LAION images that show two to ten instances of an object, 60 images per number. Each image has a caption that spells out the count. Zero-shot, the model chooses among nine captions that differ only in the number, so chance is 11 percent. CLIP ViT-B/32 is right 31.67 percent of the time, and its predictions are off by 1.53 on average.

The data explains much of it. Filtering LAION-400M for captions that state a count found over 158,000 images for "two" and only about 100 for "ten". A count is also rarely what separates a caption from the other captions in a batch, so the loss seldom asks for it. Paiss et al. fine-tuned CLIP with an extra counting loss, in which each image must prefer its true caption to a counterfactual caption with the number changed ("six dogs" for a photo of three),

Here $N$ is the number of images in a batch of counting examples, $\hat{\mathbf u}_k$ and $\hat{\mathbf v}_k$ are the unit embeddings of image $k$ and its true caption, and $\hat{\mathbf v}^{\text{CF}}_k$ is the embedding of its counterfactual caption. In plain words, each image makes a two-way choice between its true count and one wrong count. The loss was added to the CLIP loss with a weight. The CLIP loss ran on all the data, and the counting loss only on a curated set of images whose captions state counts that an object detector confirmed. CountBench accuracy rose to 75.93 percent, with zero-shot accuracy on standard benchmarks roughly unchanged. This is the NegCLIP idea again, a hard negative that differs in exactly the property you want the model to learn.

How numerals split into tokens (§2.6) matters for reading digits. CountBench spells its counts as words, though, so its failures are not about tokenization.

Spatial relations are worse

CLIP does even worse on where things are than on how many there are. Kamath et al. (2023) photographed household objects themselves for a benchmark they called What'sUp. They moved one object around another, so that each set of four photos differs only in the relation, as in a mug on, under, left of, and right of a table. There are 820 images in all. Each model chose among four captions that differ only in the preposition, so chance is 25 percent. Of the 18 models tested, CLIP ViT-B/32 scored 31.0, CLIP ViT-L/14 26.1, and NegCLIP 34.4. People were right 100 percent of the time. On their spatial sets built from COCO and from GQA, a question-answering dataset of real photos, there were two options and chance was 50. The CLIP models scored between 46 and 50.

Their study of the captions of LAION-2B, a 2-billion-pair English subset of LAION-5B, found three reasons.

• Prepositions are rare. Common spatial prepositions such as "under" or "left of" appear in only about 0.2 percent of captions. People who write alt-text leave out relations that seem obvious, such as a house "above" the water.
• Prepositions are ambiguous. "Left" can mean the viewer's left or the subject's, and "under the desk" and "under the water" look nothing alike.
• The loss rarely needs them. A batch almost never holds two captions that differ only in a preposition, the same argument as for word order.

The fixes they tried did little. Fine-tuning CLIP ViT-B/32 on 4 million LAION captions that contain prepositions moved What'sUp from 31.0 to 33.1. Fine-tuning on spatial training data built from COCO and GQA raised those two benchmarks by 14.6 points on average, yet lowered What'sUp by 4.3 points. Even then the model scored 63.9 and 59.5 percent on them, against 97.3 and 99 for people.

Pairs of images CLIP cannot tell apart

Recall the rank argument of §30.1. A dual encoder compresses each image into one vector before it sees any caption. That vector must answer, in advance, every question any caption might ask. If the loss rarely asks about counts or positions, the vector has no reason to keep them. Tong et al. (2024) turned that into a way to find the blind spots.

For each of 150 such pairs, people wrote a question about the visible difference, such as "Is the dog facing left or right?". The result is the MMVP benchmark of 300 questions. A pair counts as correct only if both its questions are answered correctly. Humans answered 95.7 percent correctly. The commercial multimodal models GPT-4V and Gemini scored 38.7 and 40.7. LLaVA-1.5, an open model that feeds a CLIP image tower's patch features to a language model, scored 24.7, below the 25 percent of guessing.

Tong et al. sorted the questions into nine visual patterns, such as orientation, counting, and position. The patterns where CLIP failed were the ones where the multimodal models failed, with Pearson correlations of 0.87 for LLaVA-1.5 and 0.71 for InstructBLIP, another open model. Scaling CLIP's model and data helped only two of the nine patterns (color and appearance, and state and condition). Orientation, counting, and position stayed hard.

So a vision-language model built on a CLIP tower inherits the tower's blind spots. Tong et al.'s fix mixes DINOv2 features into the visual tokens. Interleaving DINOv2 tokens with LLaVA-1.5's CLIP tokens raised its MMVP score from 24.7 to 28.0. §31.1 takes up the problem again. There, fusing DINOv2 and SigLIP features channel by channel, as the Prismatic models do (Karamcheti et al. 2024), gains 5 to 10 percent on localization and challenge tasks. In an interview, this is the evidence that a VLM's visual errors can start in the encoder rather than in the language model.

Social bias depends on the label set

A zero-shot classifier is a list of class names, and whoever writes the list decides what the model can say. §1.9 measured bias in word vectors with WEAT, a test of how strongly two sets of words associate with two others. A zero-shot image classifier lets us measure something more direct, the labels a model actually assigns to photos of people, and how that depends on the label set.

CLIP's authors ran such a probe (Radford et al. 2021, Section 7). They used 10,000 images from FairFace, a face dataset balanced across seven race groups, two genders, and nine age groups. They classified each image into 21 classes. There were the 14 FairFace classes (each race group crossed with "male" and "female"), four non-human classes ("animal", "gorilla", "chimpanzee", "orangutan"), and three crime-related classes ("thief", "criminal", "suspicious person").

The "child" result is the main lesson. The harm did not live only in the embeddings. It lived in how the embeddings met a label set that offered no correct label for a photo of a toddler. The softmax had to put the probability somewhere. Agarwal et al. (2021) made the same point more broadly. Because CLIP lets anyone define classes in natural language, the design of the classes can strongly shape both accuracy and how biases appear in deployment. So a bias evaluation of a zero-shot system has to evaluate label sets, not only models.

Two statistical points apply as well. Rates need intervals. And when you compare many groups and categories, you need a correction for multiple comparisons (§0.11), such as the Bonferroni correction, which divides the error rate by the number of comparisons. The cell computes both for the non-human rate.

Our Wilson interval, 4.49 to 5.34, is close to the paper's 4.6 to 5.4 but not equal to it. The paper does not say how its interval was computed. That is a reason to state the method whenever you report an interval. Corrected for 14 comparisons, the interval widens to 4.31 to 5.57.

Where does the bias come from, and does balancing the data remove it? Birhane et al. (2021) audited LAION-400M, the CLIP-filtered dataset of §30.5. They found explicit image-text pairs, including depictions of rape and pornography, racist and ethnic slurs, and malignant stereotypes. Filtering by CLIP score had not removed them, because the filter only asks whether a caption and an image match.

Alabdulmohsin et al. (2024) studied data balancing systematically. They separate two kinds of bias.

• Representation bias is how often a model associates a neutral input with one group rather than another.
• Association bias is how strongly a model links a group to an attribute, such as an occupation.

Balancing the training data reduced representation bias, and fine-tuning on balanced data countered it well. Association bias was harder. It responded far less to fine-tuning, and depended instead on how long the model trained on balanced data. Their conclusion is that balancing helps but is not sufficient on its own, and they recommend balancing from the start of training. SigLIP 2 (§30.4) adopted their de-biasing methods. In its evaluation, SigLIP L/16 at 256 pixels had a representation bias of 35.5 percent. That means it associated images of unrelated objects, such as cars, with "men" rather than "women" more than 85.5 percent of the time. SigLIP 2 of the same size and resolution had 7.3 percent.

import numpy as np

def harmful_label_rates(pred, groups, harmful):
    raise NotImplementedError

def rates_with_extra_class(scores, groups, harmful, extra):
    raise NotImplementedError

CLIP reads the text in an image

CLIP learned to read. Its training captions often contain words that also appear in the image, on signs, products, posters, and screenshots. Matching those words is an easy way to lower the contrastive loss. Goh et al. (2021) looked inside CLIP's image tower and found multimodal neurons, units that respond to a concept whether it appears as a photograph, a drawing, or the written word. A single unit might fire for photos of Spider-Man, for comic drawings of him, and for the text "spider". Because the same features serve an object and its name, writing a name on an object can move the image embedding toward the name. That is a typographic attack.

Why does the zero-shot classifier fall first, and the probe only to the stronger attack? The prompt "a photo of an iPod" was learned from captions that accompany both pictures of iPods and pictures that contain the written word. So the prompt's embedding points toward both. A probe trained on photos of unlabeled objects learns weights that point toward what the objects look like. It ignores the written-word features only to the extent that they are separate from the object features, and multimodal neurons say they are not entirely separate.

A toy with two kinds of feature makes this exact. A photo of class $k$ carries a written label for class $\ell$, and its embedding is proportional to $(1 - w)\,\mathbf o_k + w\,\mathbf t_\ell$. Here $\mathbf o_k$ is the look of class $k$, $\mathbf t_\ell$ is the look of the written name $\ell$, and $w \in [0, 1]$ is the share of the image the label takes up. The written name overlaps its own object by $\nu = \mathbf o_\ell^\top\mathbf t_\ell \ge 0$. The zero-shot prompt for class $j$ is $\mathbf o_j + \beta\,\mathbf t_j$, because web captions tie a class name to both the object and the word. The probe's weights are the $\mathbf o_k$ alone. Then the two classifiers switch from the object to the label at

In plain words, the ratio is below 1 whenever $\beta > 0$ and $\nu < 1$, so the zero-shot classifier always flips first. With $\nu = 0$ the probe flips only at $w = 1$, when the photo is all label. With any overlap, a large enough label fools the probe too.

The cell builds this toy with ten classes in 256 dimensions and checks the switch points.

With no overlap the zero-shot classifier crosses from the object to the label near the predicted 0.556, and the probe is still right 87 percent of the time at $w = 0.8$. With an overlap of 0.5 the zero-shot classifier crosses earlier, near 0.519. The probe is right 88 percent of the time at $w = 0.6$ and only 30 percent at $w = 0.7$, on either side of its predicted 0.667.

The toy is ours, a caricature with two kinds of feature, but it matches the published pattern. A weak label fools the zero-shot classifier and not the probe. A strong enough label moves the shared features so far that any classifier reading them is fooled. So the probe resists handwritten labels without making CLIP's features robust, and CLIP's notion of "iPod" is partly the shape of the letters. This is a research finding about the model's features, measured as success rates. §19.3 covers attacks and defenses in general, and sparse autoencoders find features like these systematically in language models, rather than one neuron at a time.

A short text window and other practical limits

CLIP's text tower has 77 positions, its start and end tokens included, and in practice it uses far fewer. Zhang et al. (2024) measured how retrieval with long captions improves as more of each caption is fed to CLIP. It improves only slowly after about 20 tokens, so the effective length is under 20. They attribute this to the training data. Most alt-text is short, so the position embeddings beyond the first 20 or so were rarely trained.

Their fix, Long-CLIP, keeps the first 20 position embeddings and interpolates the remaining 57 to stretch the window to 248 tokens. It then fine-tunes on about a million pairs with long captions. This improves long-text retrieval while keeping CLIP's zero-shot accuracy. The limit matters most for the recaptioned data of §30.5, whose rich synthetic captions overflow the window, and for text-to-image generation, where long prompts are common. In practice, put the words that matter first, since anything past the 77th token is cut off and much past the 20th is barely used.

Reading is uneven too. CLIP reads rendered words well, and Goh et al.'s attacks rely on it, but it reads numbers in natural photos poorly. Zero-shot CLIP transcribes the full house numbers of SVHN correctly 51 percent of the time. Its authors note that this is well below published results, and blame repeated characters and low resolution. On the handwritten digits of MNIST it reaches 88 percent, below logistic regression on raw pixels, because its training data contains almost nothing that looks like a handwritten digit.

CLIP's authors also list fine-grained classes (car models, flower species, aircraft variants), where zero-shot CLIP is poor compared with models trained for the task. On tasks unlikely to appear in web captions, such as the distance to the nearest car in a photo, it can be near random.

A softmax cannot tag two objects

The three-image example of §30.1 ended on a photo of a dog and a cat. At a logit scale of 100, the zero-shot classifier called it a cat with probability $1 - 1.7\times10^{-7}$. Calibrating the scale (§30.3) makes that answer less confident, but it cannot make it right. The defect is not the temperature. It is that the classes compete for one unit of probability.

Proof. If $m$ classes have $p_k > t$, then $1 = \sum_k p_k \ge \sum_{\text{those } m} p_k > mt$, so $m < 1/t$. $\square$

Lowering the threshold does not rescue the softmax either. The probability of "dog" is $p_{\text{dog}} = 1/\big(1 + \sum_{j\ne\text{dog}}e^{(z_j - z_{\text{dog}})/T}\big)$, so it depends on how strongly every other class is present. A dog standing next to a clearly visible cat gets a tiny probability, just as an absent dog does, and no threshold separates the two cases. Multi-label tagging, giving each image every tag that applies, needs a score for each class that does not depend on the others, a yes or no question per prompt.

Where should the threshold go? Not at one global value on CLIP's cosines, because nothing in CLIP's training fixes the absolute level of a cosine. Different prompts sit at different typical cosines, since a generic prompt matches everything a little. The modality gap lowers all cross-modal cosines together. So a cosine of 0.25 can be a confident match for one class and noise for another.

The standard remedy is a threshold for each class, fitted on a held-out set of images with multi-label annotations. A common choice is the threshold that maximizes that class's F1 score (§0.11), or the lowest one that meets a target precision. A per-class threshold on $\sigma(e^{\alpha}\cos_k + b_0)$ is the same as a per-class bias.

SigLIP is better placed (§30.4). It scores every image-text pair with its own sigmoid and was trained against one absolute boundary at logit 0, shared by every pair, so a single threshold already means something. But in training only one pair in $B$ was a match, so its bias encodes a strong prior toward "no" (it starts at $-10$). Its probabilities for class prompts therefore often come out well below one half, and per-class calibration on held-out data still pays. Audio taggers use a sigmoid per class for the same reason. On AudioSet, a clip in the evaluation set carries about 2.5 labels (§35.6).

import numpy as np

def multilabel_tags(cos, scale, bias, thresholds):
    raise NotImplementedError

def fit_thresholds(probs, labels):
    raise NotImplementedError

What the field built next

Each limit has a cause, and the fixes that work target the cause. The table sums up what was tried and what the evidence says. It is also a good answer to the interview question "what are CLIP's limitations?", because it gives a mechanism for each one.

Beyond patches to single limits, three larger moves followed.

• Objectives that keep detail. Encoders added captioning and dense objectives that force the image tower to keep detail. Examples are CoCa (Yu et al. 2022), which adds a captioning decoder to the contrastive loss, and the location-aware decoder of SigLIP 2 (§30.4).
• No more single vector. Vision-language models stopped compressing the image to one vector. They let a language model attend over all the patch tokens (§31.1), at the price of a forward pass per image and question.
• Vision without language. Self-supervised vision learning came back. Fan et al. (2025) trained visual self-supervised models and CLIP on the same 2 billion MetaCLIP images. At enough scale, up to 7 billion parameters, the self-supervised models matched CLIP on a broad set of visual question answering and classic vision benchmarks. That included OCR and chart questions, once the training images were filtered toward ones rich in text. DINOv3 (Siméoni et al. 2025) followed in the same direction.

So language is not the only route to features that a language model can use. Predicting the content of an image in latent space, rather than matching it to a caption, is the idea behind joint-embedding predictive architectures, which avoid the caption bottleneck altogether.

Images can bridge two modalities that never meet

The same objective also grew sideways, to more than two modalities. ImageBind (Girdhar et al. 2023) builds one joint space for six of them, images, text, audio, depth, thermal images, and IMU readings (accelerometer and gyroscope signals). It does so without a single dataset in which they all appear together. It freezes the image and text towers of a large CLIP (OpenCLIP ViT-H). It then trains a new tower for each other modality with this chapter's symmetric loss, against images only, on naturally paired data. The pairs are video frames and their audio from AudioSet, images and depth maps from SUN RGB-D, images and thermal images from LLVIP, and egocentric video and IMU readings from Ego4D.

Audio is never paired with text. Yet text prompts then classify sounds, because both towers were pulled toward the same images. The authors call this emergent zero-shot recognition. Why should that work, and when does it fail? The PMI theorem of §30.1 answers both questions with one identity. Recall that the pointwise mutual information of two outcomes is $\operatorname{PMI}(a, b) = \log\frac{p(a, b)}{p(a)p(b)}$, and that the contrastive loss is minimized when the score of a pair equals its PMI plus a constant.

Proof. The right side is $\sum_x p(x)\frac{p(a, x)}{p(a)p(x)}\frac{p(t, x)}{p(t)p(x)} = \sum_x p(x\mid a)\frac{p(t\mid x)}{p(t)}$. Conditional independence gives $p(t\mid x) = p(t\mid x, a)$, so the sum is $\sum_x p(x\mid a)p(t\mid x, a)/p(t) = p(t\mid a)/p(t) = e^{\operatorname{PMI}(a, t)}$. $\square$

In plain words, an audio tower trained against images learns scores that approximate $\operatorname{PMI}(a, x)$ up to a constant, and a text tower trained against images learns $\operatorname{PMI}(t, x)$. When the picture is a sufficient bridge, everything the audio says about the text passes through the picture, so the two image-anchored scores determine the audio-text score. When the sound and the caption share information that the frame does not show (the words someone speaks, a siren offscreen, the genre of the music), the identity fails. No amount of image-paired training recovers what is missing.

There is a second, smaller gap. The audio-text dot product of a dual encoder does not compute the expectation over images. It is a further approximation, which works to the extent that both towers land near the image embeddings of the same scenes. The cell checks the identity on a small discrete world, then lets the sound and the text share an offscreen cause.

With nothing shared beyond the frame, the bridge is exact. When half of what determines the sound and the text is offscreen, the bridged score already picks the wrong best text for a quarter of the sounds. When everything is offscreen it is close to guessing, with 2 of 8 sounds right against 1 of 8 by chance.

ImageBind's numbers show both sides. Emergent zero-shot accuracy on ESC-50, 50 classes of environmental sounds, is 66.9 percent. That is close to the 68.6 of AudioCLIP, which trained with audio class names as text, but far from the best reported result of 97.0, obtained with labeled audio. Clotho and AudioCaps are sets of sound clips with written captions. Text-to-audio retrieval recall at 1 is 6.0 percent on Clotho and 9.3 on AudioCaps. That is ahead of a model trained on automatically mined audio-text pairs (3.0 and 8.7), and well behind a supervised model trained on audio-text pairs (12.6 and 24.3).

The authors also find that the emergent abilities improve with the strength of the image encoder, which is the bridge argument again. And the frozen image tower passes its blind spots to every modality bound to it. Directly paired audio and text, where they exist, do better. That is the route of CLAP, a CLIP-style model trained on audio clips and their captions (§35.6).

In PyTorch

The NegCLIP loss is two cross-entropies, one over a $B\times 2B$ logit matrix and one over its first $B$ columns, transposed. Autograd also confirms both parts of the theorem that each half of the loss is blind to one mean, on raw embeddings.

Read more

• Liang et al. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. The cone effect, the embedding-shift experiment, the temperature dependence, and the effects of shifting on zero-shot accuracy and fairness.
• Thrush et al. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. The 400 examples, the three scores, and the gap between every model and people.
• Yuksekgonul et al. (2023). When and why vision-language models behave like bags-of-words, and what to do about it?. ARO, the shuffling experiment that explains why retrieval training permits bag-of-words behavior, and NegCLIP.
• Hsieh et al. (2023). SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. Blind models that beat CLIP on earlier benchmarks, and a benchmark that fixes them.
• Tong et al. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CLIP-blind pairs, MMVP, and how a CLIP tower's blind spots become a VLM's.
• Kamath et al. (2023). What's "up" with vision-language models? Investigating their struggle with spatial reasoning. Controlled photographs of spatial relations, 18 models near chance, and the LAION corpus study.
• Goh et al. (2021). Multimodal Neurons in Artificial Neural Networks. Neurons that respond to a concept in any form, and typographic attacks measured.
• Alabdulmohsin et al. (2024). CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?. Representation and association bias separated, and what data balancing does and does not fix.

Watch next

▶

▶

▶