How two encoders trained on web captions put photographs and sentences in one space, so that any phrase becomes a classifier. CLIP, SigLIP, the data that makes them work, and the exact ways they fail, taken to interview depth.

This section is about putting photographs and sentences into one space of vectors. One network turns a photo into a list of numbers, a vector. A second network turns a sentence into a vector of the same length. The two are trained together so that a photo and a sentence about the same thing point in nearly the same direction. Once that works, comparing any sentence with any photo costs one dot product.

That one cheap comparison does a great deal of work. It lets you search billions of photos by typing a phrase. It turns a list of class names into an image classifier with no training. It filters web data for later models, and it gives vision to language models and image generators. The best-known model of this kind is CLIP, short for Contrastive Language-Image Pre-training, which OpenAI released in 2021 (Radford et al. 2021).

Here is the problem it solves. Take a ResNet-50 trained on ImageNet. ResNet-50 is a 50-layer convolutional network with shortcut connections (§0.8). ImageNet is a set of 1.28 million photos, each labeled by people with one of 1,000 classes. The trained network names the main object in about 76 percent of ImageNet's 50,000 validation photos. Now show it a capybara in a raincoat and ask whether that is what it sees. It cannot answer. Its last layer has one output per class, and none of them means "capybara in a raincoat". To teach it, you would collect labeled photos, add an output, and train again. Every new question costs labeled data and a training run.

The web, meanwhile, holds billions of images that already come with text. There is alt-text written for screen readers, captions under news photos, product titles, and comments under holiday snapshots. Much of it is noise, such as "IMG_2041". Together, though, it names millions of things, from dog breeds and dishes to diagrams, memes, and the font on a street sign. So the question of this chapter is whether captions can serve as the labels.

The answer rests on three earlier pieces. The image encoders are mostly Vision Transformers (ViTs), which cut an image into patches and read them as a sequence of tokens (Chapter 27). The loss is InfoNCE, the contrastive loss of §7.2, which trains two encoders so that each query picks its own partner out of a batch. And the self-supervised encoders of Chapter 28 showed that a loss of this kind learns strong image features without labels. §29.5 already borrowed the design built here to search video with sentences. This chapter is where that design comes from.

This section sets up the answer that has shaped multimodal AI since 2021. We define the design, called a dual encoder, and the shared space it builds. We work a three-image example by hand and train a tiny model that recognizes combinations it never saw. Then we prove two facts, what its similarity score converges to and what a space of $d$ numbers can and cannot express. Last, we count what the design costs at a billion images against a model that reads image and text together. Along the way we look at the evidence that matching captions learned faster than predicting them in CLIP's experiments, and at what later work found.

A classifier can only name what its last layer lists

A supervised image classifier has two parts. The backbone, nearly all of the network, turns an image into a vector of features. The linear head multiplies that vector by a weight matrix with one column per class. That gives one score per class, and these scores are called the logits. A softmax turns the logits into probabilities that add up to one.

In symbols, the backbone maps an image to a vector $\mathbf h \in \R^{d_I}$. For ResNet-50, $d_I = 2{,}048$, the numbers left after averaging its last feature map over all positions (§0.8). The head is a matrix $\mathbf W \in \R^{d_I\times K}$, and the logits are the $K$ numbers $\mathbf W^\top\mathbf h$, one per class. For ImageNet, $K = 1{,}000$. The list of things the model can ever say is fixed on the day $\mathbf W$ is created.

Two remedies existed before 2021, and both kept the fixed list. Weak supervision trades careful labels for cheap, noisy ones and trains much larger models on them. Examples are the hashtags of 3.5 billion Instagram photos (Mahajan et al. 2018) and Google's internal JFT-300M, 300 million images with automatic labels from 18,291 classes (Sun et al. 2017). Self-supervised learning (§28.1) drops labels altogether and learns features by comparing different views of the same image. Those features name nothing, though, so every task still needs a labeled set to train a head.

Natural language gives what neither remedy did, labels that are free and a list of classes that is open. No one has to write captions for a training set, because the web already holds billions. And a caption can mention anything a person can write down. The move is to stop treating a class as a column of $\mathbf W$ and start treating it as an input. If a second network reads the class name and produces that class's column, the list of classes becomes something you type at test time.

This is the first thing an interviewer wants to hear when asking why CLIP mattered. A fixed head needs labeled images for every new class. A class written as text needs none.

Two encoders that meet in one dot product

The whole design fits in a few sentences. An image encoder turns each photo into a point. A text encoder turns each caption into a point in the same space. Training pulls each image toward its own caption and away from the other captions. After training, "a photo of a dog" lands near photos of dogs. So any sentence can search images, and any list of class names becomes a classifier.

§1.8 previewed this design at the end of its tour of encoders, and §7.2 trained it on text queries and passages. Here we make every piece precise for images and text.

Throughout this chapter $x$ is an image and $y$ is a text, such as a caption, an alt-text, or a prompt. These letters are local to this chapter.

In plain words, each tower squeezes its input to one vector of $d$ numbers, and both vectors are scaled to length one. The score says how closely the two point the same way, times a learned constant. A larger constant makes a softmax over the scores sharper.

Real widths make this concrete. The image backbone is either a ResNet or a ViT, whose tokens are square patches of the image (§27.1). The text backbone is a Transformer. The CLIP model that most later work uses, ViT-L/14, has the large ViT with 14-pixel patches. Its image backbone is 1,024 wide ($d_I = 1{,}024$), its text backbone 768 wide ($d_T = 768$), and the joint width is $d = 768$. CLIP's smaller ViT-B models use $d = 512$. §30.2 gives the exact towers and pooling.

In the attention chapters a bold $\mathbf V$ holds values. In this chapter $\hat{\mathbf V}$ always means stacked text embeddings, and the one count of attention FLOPs below is written in words.

Three images and three captions, by hand

Small numbers make the picture concrete. We work in $d = 2$ so that every vector can be drawn.

Training needs a loss that rewards the diagonal pattern. It is the loss of §7.2, recapped here. Read row $i$ of the score matrix as a guess. A softmax along the row gives each caption a probability, and the loss for image $i$ is minus the log of the probability on its own caption. This is the InfoNCE loss (van den Oord et al. 2018), with the image as the query. Read column $j$ the same way, so that each caption guesses its own image. For a batch of $B$ pairs, with image $i$ matched to caption $i$, the two halves are

So $\mathbf A$ is the softmax along each row of $\mathbf S$, and $\bar{\mathbf A}$ is the softmax down each column. CLIP's loss is the average of the two directions, and its gradient on the scores is

In the example $B = 3$. §30.2 derives this gradient from the one-direction InfoNCE gradient of §7.2. In plain words, a gradient step lowers every score in proportion to the probability the two softmaxes give it, and raises the score of each true pair by a fixed amount. A wrong pair that already gets almost no probability gets almost no push. The cell below checks the numbers of the example, then computes this loss and its gradient at two logit scales. In code, index 0 is image 1.

At scale 10 the largest gradient entry off the diagonal is 0.139, at row 3 and column 2. That is the near miss between the two-animal photo and the caption "a cat". A gradient step lowers that logit, which pushes image 3 and caption 2 apart.

At scale 100 every other wrong pair is already beaten by a wide margin. Almost all of the remaining gradient sits on that one entry (0.028) and on the two diagonal entries in its row and column ($-0.014$ each). The loss spends its effort on the pairs that are easy to confuse. That is the lesson of §7.2, that the temperature decides which negatives receive the gradient, seen in one small matrix.

The example also shows the first limit of the design. A softmax over prompts must put its probability on one class. At scale 100 it does so with absurd confidence, even when the image shows both. If an interviewer asks why CLIP's zero-shot probabilities are so extreme, point at the scale. A cosine gap of 0.16 becomes a logit gap of 16. §30.3 treats calibration, and §30.6 treats images that hold several objects.

What zero-shot means here

The classifier in the example had no trained weights of its own. Its two weight vectors were the text embeddings of the two prompts. CLIP's authors describe it in exactly these terms. It is a linear classifier followed by a softmax (a multinomial logistic regression) whose inputs and weights have length one, with no bias and with a temperature. The text tower is a network that writes the classifier's weights from the class names. A network that outputs the weights of another network is called a hypernetwork (Ha et al. 2016). CLIP's authors credit the first zero-shot image classifier of this form, with class weights predicted from text, to Lei Ba et al. (2015).

The word "zero-shot" needs care, because CLIP uses it differently from older vision papers.

CLIP's paper says plainly that it uses the term "in a broader sense" (Radford et al. 2021). It measures how well a model transfers to new tasks, not whether it has seen a concept. §30.5 shows that zero-shot accuracy on a concept tracks how often the concept appears in the pretraining data. In an interview, say "no task-specific training", never "never seen".

import numpy as np

def cross_modal_topk(U, V, k):
    raise NotImplementedError

Learning vision from text before CLIP

Researchers tried for years to learn vision from text before it worked at scale. Four steps led to CLIP, and each explains a choice CLIP kept or dropped. The first mapped images into a fixed space of word vectors. The second dropped human labels and predicted phrases from user comments. The third predicted whole captions. The fourth stopped predicting text and instead matched each image to its own text out of a batch. Then came a jump in scale. The box below gives the details and numbers of each step.

CLIP's recipe is the subject of §30.2, and its data of §30.5. It helps to see all these models as one family of objectives that learn from image-text pairs, sorted by what the image tower is asked to do.

The contrastive family has a cousin in self-supervised vision (§28.2). There, two random crops of the same image form the positive pair, so the model learns what stays the same when a photo is cropped or recolored (Chen et al. 2020a, He et al. 2020). CLIP's positive pair is a picture and a sentence. The two views differ in modality, and that is what gives its features names. The same loss also trains the two-tower recommenders of §1.7, with images in the role of users and captions in the role of items.

Why match captions instead of predicting them

If captions are the labels, the obvious objective is to predict them. You would train a captioning model and keep its image tower. CLIP started there. Its authors measured how fast each objective learned to classify ImageNet zero-shot as training went on, with the same image encoder, a ResNet-50, throughout. The measure was top-1 accuracy, the fraction of test images whose highest-scoring class is the right one.

The reason is worth being able to state in an interview. A captioner must predict the exact words that come with an image, and alt-text is wildly varied. The same beach photo could be captioned "sunset at Ko Lipe", "our honeymoon!!", or "stock photo 4471". A model that must put probability on the exact words spends most of its capacity on phrasing that has nothing to do with the picture. A bag-of-words target throws away word order, which already makes the task easier. The contrastive objective goes further. It asks only which of the $B$ captions in the batch goes with this image. Any representation that separates the right caption from the wrong ones solves it.

CLIP cites the same pattern in learning from images alone. Contrastive objectives learned better representations than their predictive counterparts (Tian et al. 2019). Generative image models needed over ten times more compute for the same representation quality (Chen et al. 2020b).

CLIP's authors also give a useful way to see one training step. A batch of 32,768 pairs is a classification problem with 32,768 classes and one example per class. The classes are written in natural language, and they change every step. So the text tower learns to write a classifier for any set of sentences, because that is all it ever does in training.

This is an empirical finding about one setup, not a law. Tschannen et al. (2023) later compared captioning and contrastive pretraining with matched data, compute, and model size. Plain captioning gave image encoders that were competitive on classification and better on vision-language tasks. The captioning family is alive, and §30.4 covers it. What contrastive training uniquely gives you is the dual encoder itself, two towers whose outputs can be compared with one dot product. That is what makes zero-shot classification and web-scale search cheap, and it is the point a strong interview answer ends on.

Zero-shot transfer in miniature

Before any theory, let's build a small world where we can watch zero-shot transfer happen. Each image shows two concepts, a color (one of 4) and a shape (one of 5), so there are 20 kinds of image. An image is a noisy sum of a color vector and a shape vector in $\R^{32}$. That is a linear stand-in for the features a real backbone would produce.

A caption is a bag of words, the count of each word of a 15-word vocabulary, with word order ignored. The vocabulary holds the 4 colors, the 5 shapes, and 6 filler words, because alt-text is messy. We train on 16 of the 20 combinations and hold out four, red star, green ring, blue cone, and yellow ball. Those four never appear in any training pair. Every color and every shape does appear in other combinations.

The dual encoder is as small as it can be. Each tower is one linear map, $\mathbf W_I \in \R^{32\times 8}$ for images and $\mathbf W_T \in \R^{15\times 8}$ for captions, followed by normalization and a fixed logit scale of 10. Training is gradient descent on the symmetric loss, and each step uses fresh noisy images of the 16 training combinations.

The gradient on the logits is the formula above. It flows back through the scale, then through the normalization, then through the two projections. The normalization step removes the part of the gradient that points along the vector itself, since changing a vector's length does not change its direction, and divides what is left by the vector's length. §30.2 derives that step.

The loss starts at 4.5. A softmax that spread its probability evenly over the 16 captions would score $\log B = \log 16 = 2.77$. The random encoder does worse than that, because at scale 10 it is confidently wrong, the effect §7.2 measured for small temperatures. The loss then falls below 0.2. Every batch has fresh noise, so the printed numbers are averages over the most recent hundred steps.

That is the label bottleneck in miniature. A text tower that reads words can build a new class from parts it knows, and a lookup table cannot.

Why does the bag-of-words tower succeed? Its embedding of "red star" is the learned vector for "red" plus the learned vector for "star", each learned from other pairs. The image tower has learned to add the matching image directions. So the sum of two known parts points at the new whole.

The same property is a weakness. A bag of words gives "a red cube on a blue ball" and "a blue cube on a red ball" the same embedding. §30.6 shows that real CLIP models behave much like this on such pairs. An interviewer who asks how CLIP can recognize something it never saw wants both halves, composition from known words and blindness to their order.

Drag the step slider back to 0. Every caption points in a random direction, and the nearest caption is rarely the right one. Press Train and watch images gather next to their captions, including the dashed pairs that were never trained on. Then switch on the class prompts. "A photo of a cube" has no color word, so it lands among the cubes of every color, as a class prompt should.

What the similarity learns to estimate

Training made matching pairs score high. What exactly does the score converge to? Here is the answer in plain words first. A perfectly trained dual encoder scores a pair by how much more often that image and that text occur together than they would if they were unrelated. That quantity has a name, and it is the same quantity that word counts led to in §1.4.

Model the data as a joint distribution $p(x, y)$ over images and texts, with marginals $p(x)$ and $p(y)$. A training batch is $B$ independent pairs $(x_i, y_i)_{i=1}^{B}$ drawn from it.

A tiny table shows why this is the right target, and how it differs from the probability a captioner predicts.

In plain words, a perfectly trained dual encoder scores a pair by its PMI. The row loss alone lets each image add its own constant to all of its scores, because a softmax along a row cannot see a constant added to the whole row. The column loss alone lets each text do the same. Reading the matrix both ways pins the score down to one global constant.

That last point matters in practice whenever you threshold a similarity, for example to decide whether a caption matches an image at all. A threshold only means the same thing for every image and every text if their scores share one constant. A retriever trained in one direction gives no such guarantee, and interviewers like to ask why.

The proof rests on one result of §7.2. InfoNCE is a guessing game. One image is shown with $B$ candidate captions, one of them its own, and the model must guess which. The best possible guesser scores each candidate by how much more likely it is under the image than in general.

Three consequences are worth carrying away.

• Popular captions get no credit for being popular. PMI divides by $p(y)$, so a caption that goes with everything ("image", "photo", "click here") scores near zero, as in the table above. The in-batch softmax learns this correction on its own, because popular captions show up as negatives in proportion to their frequency. Recommender systems want the opposite. They need $\log p(\text{item}\mid\text{user})$, which ranks popular items higher, so they subtract $\log p(\text{item})$ from each training logit and the learned dot product absorbs it. That is the sampling-frequency correction of §1.7.
• Impossible pairs want a score of minus infinity. Step outside the theorem's support assumption for a moment. If an image and a text never occur together while each occurs on its own, their PMI is $-\infty$. The loss then approaches its lowest value but never reaches it. A scaled cosine is bounded, $\lvert s\rvert \le e^{\alpha}$, so the only way to push such pairs far down is a larger scale. That is one more pressure that drives CLIP's learned scale up to its cap of 100 (§30.2).
• It is word2vec again. Skip-gram with negative sampling learns a vector for each word $w$ and each context word $c$. Their dot product is $\operatorname{PMI}(w, c) - \log k$, where $k$ is the number of negatives (§1.5). A dual encoder learns image and caption vectors whose dot product is $\operatorname{PMI}(x, y)$ plus a constant. It is the same object across two modalities, with one softmax in place of $k$ separate sigmoids.

The statement for a finite batch is exact. For exercises and for intuition it helps to let the batch grow, which turns the loss into a simple expression in $p$.

The next cell checks the theorem on a small discrete world, a joint table over 40 image types and 40 caption types. Each image type has a favorite caption type, with some shared structure on top. The function pop_loss_grad computes the large-batch loss of the derivation, averaged over both directions, and its gradient. We reuse the table for the rank question in the next part.

Two exercises turn this into code. The first builds two objects this section keeps comparing, the PMI matrix of a table of counts and its best low-rank approximation. The second computes the large-batch loss and checks the theorem.

import numpy as np

def pmi_from_counts(C, eps=0.5):
    raise NotImplementedError

def rank_d_factors(P, d):
    raise NotImplementedError

import numpy as np

def population_clip_loss(P, S):
    raise NotImplementedError

What one vector per side can express

The theorem says what the score wants to be. The architecture limits what it can be. A dual encoder squeezes each image into one vector of $d$ numbers and each text into another, and every score is one dot product. So the whole table of scores between $n$ images and $m$ texts is the product of an $n\times d$ matrix and a $d\times m$ matrix. Such a product cannot be any table you like.

Start with three images and three captions, where each image should match only its own caption. The ideal score table is the identity matrix $\mathbf I_3$, with 1 for a match and 0 otherwise. With $d = 1$, each image is a single number $u_i$ and each caption a number $v_j$. The table is then $u_iv_j$, a matrix of rank 1, meaning all its rows are multiples of one row. The closest rank-1 table to $\mathbf I_3$ keeps one of the three diagonal entries and misses the other two.

Measure the miss by the Frobenius error, the square root of the sum of squared errors over all entries. Here it is $\sqrt 2 \approx 1.41$. With $d = 2$ the error falls to 1, and with $d = 3$ the table is exact. Each extra dimension buys one more independent pattern.

The general statement uses the singular values of a matrix, $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$. Each one measures how much of the matrix lies along one of its independent directions (§0.10). The identity $\mathbf I_3$ has three singular values, all equal to 1, which is why each dropped dimension above cost exactly one unit of squared error.

Proof. Every column of $\hat{\mathbf U}\hat{\mathbf V}^\top$ is $\hat{\mathbf U}$ times a column of $\hat{\mathbf V}^\top$, so it is a combination of the $d$ columns of $\hat{\mathbf U}$. The column space therefore has dimension at most $d$, and multiplying by the positive scalar $e^{\alpha}$ does not change it.

For the inequality, note that $\mathbf S - c\,\mathbf 1\mathbf 1^\top$ is a matrix of rank at most $d$ plus one of rank at most 1, so its rank is at most $d + 1$. The Eckart-Young theorem (§0.10) says that the best approximation of rank at most $r$ in Frobenius norm keeps the top $r$ singular values, and its error is $(\sum_{k>r}\sigma_k^2)^{1/2}$. No matrix of rank at most $r$ does better. Apply it with $r = d + 1$ to $\lVert\boldsymbol\Pi - (\mathbf S - c\,\mathbf 1\mathbf 1^\top)\rVert_F$, which is the left side. ∎

In plain words, a dual encoder can only produce score tables that are sums of $d$ simple patterns. If the true association between images and texts has more structure than that, some of it is lost. The free constant buys at most one extra pattern.

Be precise about what this does and does not say. Training does not minimize Frobenius error. It minimizes the contrastive loss, which weights pairs by their probability and only cares about each score relative to its row and column. So the theorem is a floor for one error measure, a reference point, not a statement about the loss. The unit length of the embeddings restricts things further, since every score lies between $-e^{\alpha}$ and $e^{\alpha}$.

Ranking also needs much less than exact values, as one more small example shows.

The cell below makes the gap between the best table in Frobenius norm and the best table for the loss concrete, on the 40 by 40 table above. For each width $d$ it compares two score matrices of rank $d$. The first is the truncated SVD of the PMI matrix, the top $d$ singular values and vectors, which is what rank_d_factors of Exercise 30.1b returns. The second is a product of two $40\times d$ factors trained by gradient descent (with the Adam optimizer) on the large-batch CLIP loss. These factors are free, without the unit length of a real dual encoder, so the comparison isolates the effect of the loss. For each it records top-1 agreement, how often each image type's top caption matches its top caption under PMI.

So a dual encoder trained contrastively is a low-rank approximation of PMI, but not the one the SVD would give you.

Where does the bound bite in practice? Not much on ordinary retrieval, where each image only needs to beat the other candidates and orders are cheap. It bites on fine distinctions among many texts for one image. "A dog chasing a cat", "a cat chasing a dog", "two dogs and a cat", and "a dog to the left of a cat" all need different scores against the same photo. One vector per image must encode all those distinctions in advance, without knowing which question will be asked. §30.6 measures how badly real models do on exactly these. An interviewer who asks "why can't CLIP tell who chased whom?" wants this argument.

Dual encoders, fusion models, and what each query costs

The alternative to one dot product is to let image and text interact inside the network. A fusion model reads the image and the text together and outputs a match score. It usually does so with cross-attention, in which each word's vector attends to the image patches (§4.6). The retrieval chapters call such a scorer a cross-encoder (§7.2). A vision-language model is a fusion model that outputs an answer in words instead of a score. Fusion can represent any function of the pair, so it has no rank bound and can see word order. The price is that nothing can be computed before the query arrives.

Count the cost of searching $M$ images for $Q$ text queries. Write $F_I$ and $F_T$ for the FLOPs (floating-point operations) of one forward pass of the image tower and of the text tower, and $F_{\text{pair}}$ for one fusion pass.

• A dual encoder embeds the corpus once, $M F_I$, and each query once, $Q F_T$. It then scores every query against every image with $QMd$ multiply-adds, which is $2QMd$ FLOPs for an exact scan. An approximate nearest-neighbor index, which finds most of the closest vectors without comparing against all of them, cuts the scan by orders of magnitude (§7.4).
• A fusion model needs one pass per pair, $QM F_{\text{pair}}$, and can cache nothing that depends on the query.

In plain words, the dual encoder pays for the corpus once and almost nothing per query, while the fusion model pays for the whole corpus again on every query. Setting the two costs equal and solving for $Q$ gives the break-even number of queries,

beyond which the dual encoder is cheaper. If the fusion model is as large as the two towers, $F_{\text{pair}} \approx F_I + F_T$, and $Q^\star \approx F_I/(F_I + F_T)$, which is less than one query. So the dual encoder is cheaper from the first query on. Fusion can be cheaper for more than a few queries only if one fusion pass costs far less than one pass of the image tower. Then $Q^\star$ is roughly the ratio $F_I/F_{\text{pair}}$.

Move the corpus slider down to 10,000 images. The fusion line still ends far above the dual one, because the fusion model reads every image for every query. Then shrink the fusion pass to 0.1 percent of the towers, as for a small cross-attention head over cached image features. The break-even rises to about 900 queries.

The design that real systems use falls out of this. Retrieve a shortlist with a dual encoder, then rerank it with something that reads both inputs. That two-stage answer, with these numbers behind it, is what an interviewer wants when asking "dual encoder or cross-encoder?".

Late interaction, between one dot product and fusion

There is a middle ground. It is the standard answer to "how do you escape the rank bound without paying for a cross-encoder?" Keep one vector per image patch and one per text token, instead of pooling each side to a single vector. Encode the two sides separately, as before, and let them meet only in a cheap score at the very end. This is called late interaction. It was introduced for text retrieval (Khattab and Zaharia 2020, a model called ColBERT, §7.3) and brought to image-text pretraining a year later (Yao et al. 2021, a model called FILIP).

In plain words, each word looks for the one patch that matches it best, and the score is how well the words did on average. So "red" can match the red patches and "cube" the cube.

FILIP uses these averages. ColBERT sums over the query tokens instead, and so does ColPali, which applies the same idea to images of document pages (Faysse et al. 2024). For one query, the sum and the average rank the candidates identically, since $n_y$ is fixed. FILIP chose the average because the number of tokens varies from text to text. Sums over different lengths have very different sizes, which FILIP found made training less stable and the final model worse.

The score is no longer one dot product $\hat{\mathbf u}^\top\hat{\mathbf v}$. A maximum over patches is not linear in either side, so the matrix of late-interaction scores is not confined to rank $d$, and the rank bound does not apply. The cell below checks this with random unit vectors in $d = 4$. Pooled to one vector per side, 40 images and 40 texts give a score matrix of rank 4, as the theorem requires. Kept as 8 patches and 5 tokens each and scored by late interaction, the same width gives a matrix of full rank 40.

The patch vectors are still computed without seeing any text, so the corpus is encoded once, exactly as for a dual encoder. What late interaction does not buy is reasoning about relations. Each token picks its best patch on its own. So "a dog chasing a cat" and "a cat chasing a dog" score differently only if their token vectors differ. They differ somewhat, since each token vector has seen its neighbors in the sentence, but nothing in the score compares who is doing what to whom.

The table puts the four designs side by side.

def retrieval_budget(M, Q, flops_img, flops_txt, flops_fusion_pair, d, bits_per_dim):
    return {"dual_flops": 0.0, "fusion_flops": 0.0, "index_bytes": 0.0, "break_even_queries": 0.0}

The table and the exercise give the shape of a modern image search or tagging system. A dual encoder embeds the corpus offline and serves the first stage. An approximate index keeps the scan cheap. A fusion model or a vision-language model rescores the top few dozen candidates, where word order, counting, and relations matter.

The same dual encoders reappear far beyond search. Their image towers feed vision-language models, and their text towers condition image generators. The same objective aligns video with text and audio with text.

In PyTorch

Here is the same world of colors and shapes in PyTorch, with a deeper image tower, a small network with one hidden layer. The text tower is a bag of words. In mean mode nn.EmbeddingBag averages the learned vectors of a caption's words, which is a linear map of its word counts divided by its length. Both towers end in linear projections and unit normalization. The log of the logit scale is learned, and the scale is capped at 100, as in CLIP. The loss is cross-entropy over the rows plus cross-entropy over the columns. Retrieval in both directions is topk along a row or down a column.

The deeper image tower recognizes 85 percent of the held-out combinations. Replace it with a single nn.Linear(D_in, 64), keep everything else, and the script reaches 96 percent, although its training loss ends higher, 0.10 against 0.05. The deeper tower fits the training pairs better and composes worse. A hidden layer can learn features specific to each seen combination instead of the additive structure. So composition is not guaranteed by the loss. The data and the architecture have to support it, a point §30.5 and §30.6 develop at scale.

Read more

• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. The CLIP paper. Its comparison of pretraining objectives (Figure 2), its definition of zero-shot transfer, and its comparison with Visual N-Grams are the background of this section.
• Frome et al. (2013). DeViSE: A Deep Visual-Semantic Embedding Model. Images mapped into a skip-gram word space, an early joint image and word embedding that could score labels it was never trained on.
• Li et al. (2017). Learning Visual N-Grams from Web Data. Phrase prediction from Flickr comments and the first zero-shot transfer to ImageNet, at 11.5 percent.
• Desai & Johnson (2021). VirTex: Learning Visual Representations from Textual Annotations. Captioning as pretraining, the predictive approach CLIP started from.
• Zhang et al. (2020). Contrastive Learning of Medical Visual Representations from Paired Images and Text. ConVIRT, the two-way image-text contrastive loss that CLIP simplified and scaled.
• Jia et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. ALIGN, the same dual-encoder idea on 1.8 billion lightly filtered alt-text pairs.
• Yao et al. (2021). FILIP: Fine-grained Interactive Language-Image Pre-Training. Late interaction for image-text pretraining, with each token's best-match similarity averaged over tokens and the efficiency tricks that make it trainable.
• Sohn (2016). Improved Deep Metric Learning with Multi-class N-pair Loss Objective. The batch construction in which every other pair serves as a negative.
• Tschannen et al. (2023). Image Captioners Are Scalable Vision Learners Too. A controlled comparison showing that captioning pretraining is competitive when compute and data are matched.
• Lei Ba et al. (2015). Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions. The first zero-shot image classifier whose class weights are written by a network reading text, the form CLIP's zero-shot classifier takes.

Watch next

▶

▶