At the end of §7.1 we hit a wall. A pretrained BERT knows a great deal about language, but its vectors are bad at the one thing a search engine needs, putting texts that mean the same thing close together. What we do have is an enormous supply of pairs that should be close. A question and its accepted answer. A title and its article. A search query and the page people clicked. Two paraphrases. How do we turn pairs into a training signal for geometry?

The obvious idea is to pull each pair together. That fails in a way that's worth seeing, and the fix (push everything else apart, in a very particular way) gives us InfoNCE, the loss behind DPR, SimCSE, CLIP, SimCLR, the sentence-transformers library, and every modern embedding model. By the end of this section you'll have derived it, proved what it estimates, implemented its gradient and its symmetric CLIP form, and know exactly what the temperature does, including what happens when the model learns it.

Pulling pairs together is not enough

Suppose an encoder $f_\theta$ maps each text $x$ to a unit vector $\hat{\mathbf{z}} = f_\theta(x)/\lVert f_\theta(x)\rVert \in \mathbb{R}^d$, and we minimize the average squared distance between the two members of each positive pair $(x, x^+)$,

There's a perfect solution, and it's useless. Map every input to the same vector. The loss is zero, and every query is equally similar to every document. This is called representation collapse, and any objective that only pulls must somehow forbid it.

The classical fix is the triplet loss with a margin $\varepsilon$, which Sentence-BERT (Reimers & Gurevych, 2019) used with $\varepsilon = 1$. For an anchor $x$, a positive $x^+$, and one negative $x^-$,

In plain words, the positive must be closer than the negative by at least the margin. It works, but each example teaches the model about a single negative, and once a negative is beaten by the margin it contributes no gradient at all. We'd like every example to contrast against many negatives at once, and to weigh them by how confusable they are. That's what InfoNCE does.

InfoNCE turns similarity into classification

Take a batch of $B$ positive pairs $(q_i, p_i)$, for example $B$ questions and their answer passages. Encode and normalize everything, $\hat{\mathbf{q}}_i, \hat{\mathbf{p}}_i \in \mathbb{R}^d$, and form the similarity logits

where $\tau > 0$ is the temperature. Now treat each row as a $B$-way classification problem. "Which of these $B$ passages is mine?" The correct answer for row $i$ is column $i$, so the loss is the cross-entropy with the diagonal as the target,

In plain words, each query should score its own passage higher than every other passage in the batch, and the loss is how surprised a softmax over those scores is by the right answer.

The other $B - 1$ passages in the batch are called in-batch negatives, and they're the reason the loss is so practical. We already had to encode them as positives for their own queries, so the negatives cost nothing extra except the $B^2$ dot products in $\hat{\mathbf{Q}}\hat{\mathbf{P}}^\top$, which is tiny next to running the encoder. A batch of 1,024 pairs gives every query 1,023 negatives for free.

Let's work one by hand, small enough to check on paper. Two queries and two passages have cosines

Row 1 gives its own passage the probability $e^{1.6}/(e^{1.6} + e^{0.4}) = 1/(1 + e^{-1.2}) = 0.769$, a loss of $-\log 0.769 = 0.263$. Row 2 gives $1/(1 + e^{-0.4}) = 0.599$, a loss of $0.513$. InfoNCE is the mean, $0.388$. With two candidates, each row's softmax is a sigmoid of the gap between the positive logit and the negative one, so the loss depends only on differences of cosines, magnified by $1/\tau$. Query 2 is the weak one, because its positive beats its negative by only $0.2$ in cosine.

You've seen this loss twice before. It's the row half of the CLIP loss sketched in §1.8 (CLIP averages it with the same loss on columns, which we derive in full below), and it's a softmax version of word2vec's negative sampling from §1.5. The name comes from van den Oord, Li and Vinyals (2018), who introduced it for contrastive predictive coding. The "NCE" refers to noise-contrastive estimation, and the "Info" to a link with mutual information that we'll prove shortly.

A quick sanity check gives two numbers worth memorizing. With a random encoder and a temperature near 1, all logits are roughly equal, so every row's softmax is close to uniform and the loss is about $\log B$. A perfect encoder drives it toward zero. With $B = 256$, training starts near $\log 256 = 5.55$.

At $\tau = 1$ the random encoder sits right at $\log B$, but notice that even identical pairs can't get the loss anywhere near zero. At $\tau = 0.05$ the identical pairs reach zero, while random pairs land well above $\log B$. Their cosines spread by about $1/\sqrt{d} = 0.125$ (the high-dimensional fact from §1.3), the temperature magnifies that spread twentyfold, and a confidently wrong softmax costs more than a uniform one. We'll come back to both effects when we study the temperature.

The gradient is observed minus expected

Let's derive the gradient, because it tells you what the loss does to the embeddings, and because "implement InfoNCE with its backward pass" is a common interview question. Write $\mathbf{A} = \operatorname{softmax}(\mathbf{S})$ applied to each row, so $A_{ij}$ is the probability the model assigns to passage $j$ for query $i$. From the softmax cross-entropy gradient (§0.2),

Each $s_{ij}$ depends on $\hat{\mathbf{q}}_i$ and $\hat{\mathbf{p}}_j$ through $\hat{\mathbf{q}}_i^\top\hat{\mathbf{p}}_j/\tau$, so the chain rule gives

In matrix form, with $\mathbf{G} = (\mathbf{A} - \mathbf{I})/(B\tau)$, these are $\nabla_{\hat{\mathbf{Q}}}\mathcal{L} = \mathbf{G}\hat{\mathbf{P}}$ and $\nabla_{\hat{\mathbf{P}}}\mathcal{L} = \mathbf{G}^\top\hat{\mathbf{Q}}$.

In the hand example $B\tau = 1$, so $\mathbf{G} = \mathbf{A} - \mathbf{I}$, with rows $(-0.231,\ 0.231)$ and $(0.401,\ -0.401)$. Every row sums to zero, because a softmax row sums to one. And the weak query 2 gets the larger entries, so the gradient spends its effort where the ranking is least certain.

In plain words, a gradient step moves each query toward its own passage and away from the softmax-weighted average of all passages in the batch, the "observed minus expected" pattern of word2vec. Each passage gets pushes from every query, which is easy to forget in an implementation.

One more step. The vectors were normalized, $\hat{\mathbf{q}} = \mathbf{q}/\lVert\mathbf{q}\rVert$, and the Jacobian of normalization is

In plain words, normalization removes the part of the gradient that would only change the vector's length, since length doesn't affect the loss. The gradient with respect to an unnormalized vector is always perpendicular to it.

import numpy as np

def infonce_grad(Q, P, tau):
    """Q, P: (B, d) unnormalized. Returns (loss, dQ, dP)."""
    raise NotImplementedError

What InfoNCE estimates, a lower bound on mutual information

Why would classifying "which passage is mine" produce good embeddings? Here's the precise answer. Minimizing InfoNCE maximizes a lower bound on the mutual information between the two views,

where $N$ is the number of candidates in each classification (here $N = B$) and mutual information (§0.5) is $I(X;Y) = \mathbb{E}_{p(x,y)}\big[\log \frac{p(x \mid y)}{p(x)}\big]$, the expected log of how much seeing $y$ raises the probability of $x$. This bound is a theorem, not a heuristic, and we'll prove it in full because the proof explains three practical facts at once.

Set up the classification game precisely. Draw a context $y \sim p(y)$ (the query). Pick a secret index $i^\star$ uniformly from $[N]$. Draw $x_{i^\star} \sim p(x \mid y)$ (the true passage) and every other $x_j \sim p(x)$ independently (passages unrelated to $y$). A model with a score function $s(x, y)$, also called the critic, guesses the index with $q(i \mid X, y) = e^{s(x_i, y)} / \sum_k e^{s(x_k, y)}$, and $\mathcal{L}_N = \mathbb{E}[-\log q(i^\star \mid X, y)]$ is exactly InfoNCE.

Three consequences follow, and all three show up in practice.

1. The optimal critic is a log density ratio. A trained bi-encoder's score $\hat{\mathbf{q}}^\top\hat{\mathbf{p}}/\tau$ is trying to approximate $\log \frac{p(\text{passage} \mid \text{query})}{p(\text{passage})}$ plus a constant per query (which the softmax over candidates ignores), and that log ratio is exactly the pointwise mutual information of §1.4. Word2vec factorized PMI implicitly (§1.5), and so does a contrastive embedder.
2. The bound can never exceed $\log N$, since $\mathcal{L}_N \ge 0$. With a batch of 512, InfoNCE can certify at most $\log 512 = 6.2$ nats of shared information, however good the encoder. This is one clean reason larger batches help.
3. Negatives must come from $p(x)$. The proof assumed the other candidates are independent draws from the marginal. In-batch negatives are approximately that. Hard negatives, which we'll add soon, deliberately violate it. That's fine for learning embeddings (we care about the ranking, not the MI estimate), but it means you shouldn't read the loss as an MI estimate once you mine negatives.

Let's watch the bound behave with a case where the true mutual information is known. If $\mathbf{x}$ is standard Gaussian in $D$ dimensions and $\mathbf{y} = \rho\mathbf{x} + \sqrt{1-\rho^2}\,\boldsymbol{\epsilon}$, then $I = -\frac{D}{2}\log(1-\rho^2)$, and we can use the optimal critic exactly.

With little shared information (0.83 nats), a few dozen candidates already make the bound nearly tight. With a lot (6.64 nats), the estimate is pinned just under the $\log N$ ceiling for small $N$, and even at $N = 512$ it reaches only 5.40, short of both the ceiling (6.24) and the truth. The bound is a Monte Carlo average, so expect wobble in the second decimal. The lesson for practice is that when two views share a lot of information, a small batch can't express it, and that's exactly the regime of text retrieval, where a query pins down its answer very precisely.

import numpy as np

def nce_mi_estimate(S):
    raise NotImplementedError

Temperature decides which negatives matter

The temperature looks like a minor knob. It isn't. It controls both how confident the model is allowed to be and which negatives get the gradient.

Confidence. With unit vectors, every cosine lies in $[-1, 1]$, so every logit lies in $[-1/\tau, 1/\tau]$. At $\tau = 1$, even a perfect encoder (positive cosine 1, every negative cosine $-1$) can give its positive at most $e^{1}/(e^{1} + 255\,e^{-1}) \approx 2.8$ percent probability in a batch of 256. The loss can't go below about 3.6, and the gradient never stops pulling. A small temperature removes that ceiling. Typical values are 0.05 (SimCSE), 0.01 (E5 and GTE), 0.02 (the default in BGE's public fine-tuning scripts), and CLIP learns $\tau$ starting from 0.07.

Which negatives. Look again at the gradient on query $i$. Negative $j$ pushes it with weight $A_{ij}/\tau$, where $A_{ij} \propto e^{s_{ij}} = e^{\cos_{ij}/\tau}$. At small $\tau$, the softmax concentrates almost all of that weight on the few negatives with the highest cosine, the hard negatives. At large $\tau$, the weights flatten and every negative gets a similar push. Wang and Liu (2021) call this the hardness-aware property of the contrastive loss, and they show a trade-off. Low temperatures spread embeddings more uniformly but become intolerant of near-duplicates, pushing apart negatives that are actually semantically close.

Hard negatives and bigger batches

Random in-batch negatives quickly become easy. Once the model can tell a question about tax law from a passage about volcanoes, those negatives give almost no gradient, which is the temperature argument again. Two levers keep the task hard.

Hard negatives. Add, for each query, passages that look relevant but aren't. Dense Passage Retrieval (DPR, Karpukhin et al., 2020) added one BM25-retrieved passage per question that matched many question words but didn't contain the answer, and shared all of them across the batch, so each query saw $B - 1$ in-batch negatives plus $B$ hard negatives. Later work mined negatives with the retriever itself as it trained (ANCE, Xiong et al., 2021), or with a stronger cross-encoder. With hard negatives, the logit matrix becomes $B \times (B + Bk)$ for $k$ hard negatives per query.

Bigger batches. More negatives raise the $\log N$ ceiling and make it likelier that a few hard ones show up by chance. In E5's ablation (Wang et al., 2022), growing the contrastive pretraining batch from 1K to 8K to 32K raised the average nDCG@10 on six BEIR datasets from 45.8 to 50.2 to 51.6. Since encoder activations dominate memory, big batches are built with tricks. You can all-gather embeddings across GPUs so every device scores against the whole global batch (§10.2), cache gradients with respect to embeddings and re-run the encoder in small chunks (GradCache, Gao et al., 2021), or keep a queue of recent embeddings as extra negatives, an idea MoCo popularized for images.

import numpy as np

def hard_negative_loss(Q, P, Nh, tau, pos_ids, neg_ids):
    raise NotImplementedError

The symmetric loss and a learned temperature, as in CLIP

Everything so far scored one direction. Each query picks its passage out of the batch. CLIP (Radford et al., 2021), which §1.8 sketched, trains an image encoder and a text encoder together, and it needs both directions at test time, finding the caption for an image and finding the image for a caption. So it scores both. Let the rows of $\hat{\mathbf{Q}}$ be the $B$ normalized image embeddings and the rows of $\hat{\mathbf{P}}$ the $B$ normalized caption embeddings, with the same logits $s_{ij} = \hat{\mathbf{q}}_i^\top\hat{\mathbf{p}}_j/\tau$. The row loss is InfoNCE as before. The column loss asks each caption $j$ to pick its own image out of the $B$ images,

In plain words, the same $B \times B$ logit matrix is read twice, once as $B$ classification problems over its rows and once over its columns, and the two cross-entropies are averaged. The only difference between the directions is where the softmax normalizes.

Back to the hand example. Column 1 compares $1.6$ with $0.8$, so caption 1 picks image 1 with probability $1/(1 + e^{-0.8}) = 0.690$ and loss $0.371$. Column 2 compares $1.2$ with $0.4$, the same gap, so it has the same loss. $\mathcal{L}_{\text{col}} = 0.371$, and $\mathcal{L}_{\text{CLIP}} = (0.388 + 0.371)/2 = 0.380$. The two views disagree about what is hard. In the rows, query 2 is the weak one, with a cosine gap of 0.2. In the columns, both captions have a gap of 0.4, because each caption is compared with the other image's cosine to it (0.4 for caption 1, 0.2 for caption 2).

The gradient. Write $\bar{\mathbf{A}}$ for the softmax taken down each column, $\bar A_{ij} = e^{s_{ij}}/\sum_{i'} e^{s_{i'j}}$. The column loss is a cross-entropy over column $j$, so the same "p minus y" rule gives $\partial\mathcal{L}_{\text{col}}/\partial s_{ij} = (\bar A_{ij} - \mathbb{1}[i = j])/B$. Adding the halves,

In plain words, it is the one-directional gradient with the row softmax replaced by the average of the row and column softmaxes. Rows of $\mathbf{G}_{\text{CLIP}}$ no longer sum to zero, but rows of $\mathbf{A} - \mathbf{I}$ and columns of $\bar{\mathbf{A}} - \mathbf{I}$ still do, which is a quick test for an implementation. The normalization Jacobian from above then carries these gradients back to the raw embeddings unchanged.

Learning the temperature. CLIP doesn't tune $\tau$ by hand. It learns a scalar $\alpha$ with $1/\tau = e^{\alpha}$, so $s_{ij} = e^{\alpha}\,\hat{\mathbf{q}}_i^\top\hat{\mathbf{p}}_j$. The exponential keeps the scale positive, and it makes the derivative clean, $\partial s_{ij}/\partial \alpha = e^{\alpha}\,\hat{\mathbf{q}}_i^\top\hat{\mathbf{p}}_j = s_{ij}$. By the chain rule, $\partial\mathcal{L}/\partial \alpha = \sum_{i,j} (\partial\mathcal{L}/\partial s_{ij})\, s_{ij}$. Split it into the two halves and use $\sum_j A_{ij} = 1$,

In plain words, the temperature's gradient is how far the softmax-weighted average logit sits from the positive logit, averaged over rows and columns. When positives score above that average, which is what a working encoder does, the gradient is negative, so gradient descent raises $\alpha$ and sharpens the softmax. In the hand example the row terms are $-0.277$ and $-0.160$ and both column terms are $-0.248$, so $\partial\mathcal{L}/\partial \alpha = -0.234$.

That has a sharp consequence. If every positive is the largest logit in its row and its column, a larger scale always lowers the loss, all the way to zero, so the learned temperature runs off toward zero and the logits blow up. Only the mistakes push back, because a row whose positive is beaten pays more as the softmax sharpens. So the learned $\tau$ settles where the model's confidence matches its error rate, and CLIP adds a guard. It starts at $\tau = 0.07$ and clips the scale $e^{\alpha}$ at 100, which means $\tau \ge 0.01$. Watch the loss as a function of the scale for three batches of different difficulty.

The clean batch, where every image already finds its caption, keeps getting cheaper as the scale grows and would push the temperature to zero without the clip. The middle batch has its best scale near 30, where its 7 percent of mistakes start to cost more than its confident hits save. The hard batch wants a small scale, because a confident softmax is mostly confidently wrong. A learned temperature tracks this as training makes the task easier, and the clip at 100 is what stops it in the late, nearly separable phase.

At real scale. CLIP trained on 400 million image and caption pairs with a batch of 32,768. Its logit matrix has $32{,}768^2 \approx 1.07 \times 10^9$ entries, 4.3 GB in float32, and the two softmaxes and their gradients need several times that. Nobody materializes it in one place. Each GPU gathers every embedding in the global batch (they're small, $32{,}768 \times d$), computes only the rows for its own images and the columns for its own captions, and its share of the matrix shrinks by the number of GPUs. With 256 GPUs, each holds two slices of $128 \times 32{,}768$, about 34 MB.

A different fix avoids the batch-wide softmax altogether. SigLIP (Zhai et al., 2023) scores every entry of the matrix as its own binary problem, $-\frac{1}{B}\sum_{i,j}\log\sigma\big(z_{ij}(e^{\alpha}\hat{\mathbf{q}}_i^\top\hat{\mathbf{p}}_j + b_0)\big)$ with $z_{ii} = 1$ and $z_{ij} = -1$ off the diagonal, a learned bias $b_0$ (initialized at $-10$, since each positive faces $B - 1$ negatives), and $e^{\alpha}$ initialized at 10. Every term needs only its own pair, so no row or column sum has to be gathered, which is the story of the sigmoid losses that followed CLIP.

import numpy as np

def clip_loss_grad(Qn, Pn, t, max_scale=100.0):
    """Qn, Pn: (B, d) unit rows. t: log logit scale. Returns (loss, dQn, dPn, dt)."""
    raise NotImplementedError

Sentence-BERT and the bi-encoder

Now we can say exactly what Sentence-BERT (SBERT, Reimers & Gurevych, 2019) contributed. It ran BERT on each sentence separately, mean-pooled the token vectors into one embedding (mean beat [CLS] in their ablations), and fine-tuned the shared encoder on pairs so that cosine similarity became meaningful. The architecture with two weight-sharing towers is called a siamese network, and the general pattern, where query and document are embedded independently and compared by a dot product, is called a bi-encoder.

The paper's motivating number is worth reproducing. To find the most similar pair among 10,000 sentences, a cross-encoder (BERT reading both sentences together) needs a forward pass for every pair.

SBERT measured about 5 seconds for the embeddings, against 65 hours for the cross-encoder. That gap is the reason bi-encoders exist. A bi-encoder can embed a corpus once, offline, and answer each query with one forward pass plus a nearest-neighbor search (§7.4). The price is that query and document never see each other inside the network, so subtle interactions ("not", word order, which entity does what) are harder to capture. We'll recover some of that accuracy with cross-encoder reranking in §7.5.

SBERT itself trained on natural language inference data (570,000 SNLI pairs and 430,000 MultiNLI pairs) with a softmax classifier over the features $(\mathbf{u}, \mathbf{v}, |\mathbf{u}-\mathbf{v}|)$. The sentence-transformers library that grew from it soon made in-batch InfoNCE, under the name multiple negatives ranking loss, the default, because it works better for retrieval. SimCSE (Gao, Yao & Chen, 2021) then showed two neat things with InfoNCE at $\tau = 0.05$. Without any labels, you can make a positive pair by passing the same sentence through the encoder twice, because dropout makes the two embeddings slightly different. With NLI labels, entailment pairs make good positives, and contradictions make good hard negatives.

Alignment and uniformity

What does a good embedding space look like? Wang and Isola (2020) gave a clean answer with two measurable properties, for unit vectors on the hypersphere.

Alignment. Positive pairs should be close,

Uniformity. The embeddings of all inputs should spread out over the sphere, measured by a log average Gaussian potential between random pairs, with $t = 2$ by default,

In plain words, alignment asks that pairs agree, uniformity asks that everything else disagree, and lower is better for both. A collapsed encoder has perfect alignment (0) and the worst possible uniformity (0, since every distance is zero). The uniform distribution on the sphere minimizes uniformity.

The link to InfoNCE is a theorem. As the number of negatives $M$ grows, InfoNCE minus $\log M$ converges to an alignment term plus a uniformity-style term,

So the temperature is literally the width of the repulsion kernel. Small $\tau$ means a large $t$, a sharp kernel, and repulsion dominated by each point's nearest neighbors, exactly the hard-negative focus we found from the gradient.

This also explains a puzzle from §1.9. Raw BERT and GPT-2 vectors are anisotropic, crowded into a narrow cone where random pairs have high cosine. That's a terrible uniformity score. Contrastive training fixes it directly, and SimCSE showed its gains on semantic similarity come with much better uniformity at nearly unchanged alignment.

Let's verify the whole story on synthetic data. Each example has a hidden 6-dimensional "meaning", and we observe two noisy 32-dimensional views of it. We train a linear encoder (with a bias, so collapse is possible) twice, once with InfoNCE and once with only the pulling loss.

The pull-only encoder reaches a perfect alignment score and a uniformity of zero, which is collapse. It learned to make its bias huge, so every input lands on the same point, and its recall@1 is that of a blind guess (1 in 500). InfoNCE pays a little alignment for a lot of uniformity, and that's what makes retrieval work.

import numpy as np

def alignment_uniformity(Za, Zb, t=2.0):
    raise NotImplementedError

In PyTorch

In PyTorch, InfoNCE is one call to cross_entropy with arange targets. Here it is with a learnable temperature (parameterized as CLIP does, by its log inverse, and clamped), shared hard negatives, and duplicate masking.

To get a global batch across GPUs, gather the embeddings before computing the logits. torch.distributed.nn.functional.all_gather keeps the gradient flowing back to each device's own encoder, which the plain all_gather does not.

The last row matters for multimodal models. Replacing the batch softmax with an independent sigmoid on each pair removes the need for a global normalizer, so the batch size no longer changes what the loss means, which is the idea behind the sigmoid loss of SigLIP. And contrastive learning for images, where the two views are random crops and color changes of one photo, follows the same mathematics in SimCLR and MoCo.

Read more

• van den Oord, Li & Vinyals (2018). Representation Learning with Contrastive Predictive Coding. The InfoNCE loss and its link to mutual information.
• Poole et al. (2019). On Variational Bounds of Mutual Information. A rigorous treatment of the InfoNCE bound and its log N ceiling.
• Wang & Isola (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. The two properties, their metrics, and the asymptotic theorem.
• Wang & Liu (2021). Understanding the Behaviour of Contrastive Loss. Temperature as a hardness knob, and the uniformity-tolerance dilemma.
• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). The symmetric loss, the learned and clipped temperature, and the sharded similarity matrix at a batch of 32,768.
• Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Siamese encoders, pooling ablations, and the 65 hours versus 5 seconds comparison.
• Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. In-batch negatives plus BM25 hard negatives, beating BM25 by a wide margin.
• Gao, Yao & Chen (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. Dropout as augmentation, NLI hard negatives, and the uniformity analysis.

Watch next

▶

▶

▶