Chapter 30 · Section 30.4 · 100 min

SigLIP and better losses

This section is about a second way to train an image encoder and a text encoder together, with a simpler loss. CLIP, the model of the previous sections, asks each image to pick its own caption out of all the captions in the batch, a multiple-choice question. The sigmoid loss instead asks one yes-or-no question about every image and every caption in the batch. Do these two belong together? That one change makes training on many devices cheaper, works better at small batches, and gives each score a meaning on its own. Models trained this way, SigLIP and SigLIP 2, are the image encoders inside many open vision-language models, PaliGemma and Gemma 3 among them, so interviewers expect you to know the loss exactly.

Here is the setup of §30.2 in brief. A batch holds $B$ matching pairs, each an image and its caption. An image encoder and a text encoder, called the two towers, turn each input into a vector of the same width. Both vectors are scaled to length one, so their dot product is their cosine. Each cosine times a learned scale is a logit, a raw score that a softmax or a sigmoid turns into a probability. The logits of every image with every caption fill a $B\times B$ matrix, with the matching pairs on its diagonal. CLIP's loss reads each row as a softmax classification of one image over the $B$ captions, and each column as one of a caption over the $B$ images (§7.2).

A softmax divides by a sum over its whole row, called its normalizer, and that has three costs. First, every logit in a row is tied to every other, so adding an unrelated pair to the batch changes the loss of every pair already there. Second, the normalizer needs a whole row and a whole column in one place. Training on many devices must therefore copy every embedding onto every device. Third, the loss means something different at each batch size. It starts near $\log B$ . And §7.2 showed that it can certify at most $\log B$ of mutual information, the measure of how much an image tells you about its caption.

The sigmoid loss of Zhai et al. (2023), called SigLIP (sigmoid loss for language-image pretraining), removes the normalizer. It asks of every entry of the matrix whether it is a real pair or a fake one, and scores each answer with logistic regression. §7.2 previewed this loss in one line, with its starting bias of $-10$ and scale of 10, and left the reasons for this section.

The first half of the section builds the loss and proves what it does. We derive the loss and its gradient, and compare how hard each wrong caption is pushed under the sigmoid and under the softmax. We prove why the bias must start near $-\log B$ , and say exactly what it means that the loss "decouples the batch size". Then we build the ring computation that never gathers the batch. The second half covers other ways to get better encoders. They are freezing a pretrained image tower, adding a captioning loss, combining both as SigLIP 2 does, and distilling a large encoder into a small one. Near its end the section surveys the encoders that compete with SigLIP 2 as of late 2026, then closes with when to choose each loss.

One yes-or-no question per pair

Here is the idea before the symbols. A batch of $B$ image-caption pairs offers $B\times B$ ways to put an image next to a caption. Exactly $B$ of them are real pairs. The rest are fakes, made by mismatching. Treat each of the $B^2$ combinations as its own yes-or-no question, "is this a real pair?". Score each answer with the log-loss of logistic regression, the loss of a classifier that outputs one probability. No question depends on any other.

Now the symbols. A batch holds $B$ matching pairs $(x_i, y_i)$ for $i = 1,\dots,B$ , an image $x_i$ and its caption $y_i$ . The image tower ends in a pooled feature vector $\mathbf h^I_i$ , and a projection matrix $\mathbf W_I$ maps it into the shared space, $\mathbf u_i = \mathbf W_I^\top\mathbf h^I_i \in \R^d$ . Here $d$ is the width of the shared space. The text tower likewise gives $\mathbf v_j = \mathbf W_T^\top\mathbf h^T_j \in \R^d$ . Their unit-length versions are $\hat{\mathbf u}_i = \mathbf u_i/\lVert\mathbf u_i\rVert$ and $\hat{\mathbf v}_j = \mathbf v_j/\lVert\mathbf v_j\rVert$ , exactly as in §30.2. Stacked as rows, they form $\hat{\mathbf U}, \hat{\mathbf V} \in \R^{B\times d}$ . In this chapter $\hat{\mathbf V}$ holds text embeddings, never attention values. Throughout, $\sigma(t) = 1/(1 + e^{-t})$ is the logistic sigmoid of §0.4, which turns any real number into a probability.

📐 The SigLIP loss

For a batch of $B$ pairs with unit embeddings $\hat{\mathbf u}_i$ , $\hat{\mathbf v}_j \in \R^d$ , a learned log scale $\alpha$ , and a learned bias $b_0$ , define the logits and the labels

s_{ij} = e^{\alpha}\,\hat{\mathbf u}_i^\top\hat{\mathbf v}_j + b_0, \qquad z_{ij} = 2\,\mathbb 1[i = j] - 1 \in \{+1, -1\}, \qquad i, j = 1,\dots,B .

The scale is stored as its logarithm $\alpha$ , so that $e^{\alpha}$ stays positive. The model says that pair $(i, j)$ matches with probability $\sigma(s_{ij})$ , so the probability it gives the true label is $\sigma(z_{ij}s_{ij})$ (using $1 - \sigma(t) = \sigma(-t)$ ). The SigLIP loss is the negative log-likelihood of all $B^2$ labels, treated as independent, divided by $B$ ,

\mathcal L_{\text{sig}} = -\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B}\log\sigma(z_{ij}s_{ij}) = \frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B}\log\big(1 + e^{-z_{ij}s_{ij}}\big).

SigLIP starts training at $e^{\alpha} = 10$ and $b_0 = -10$ . CLIP starts its scale at $1/0.07 \approx 14.3$ and has no bias.

In plain words, every entry of the logit matrix is its own yes-or-no question. The diagonal entries should answer yes and all the others no. The loss is the total log-loss of those $B^2$ answers, divided by the number of images. We call a matching pair a positive and a mismatched one a negative.

Three details of the definition each have a reason.

Why divide by $B$ and not by $B^2$ . Write the loss as $\frac1B\sum_i \ell_i$ , where $\ell_i = \sum_j \log(1 + e^{-z_{ij}s_{ij}})$ is one image's loss against all $B$ captions. The loss is then an average over images, like CLIP's, and this is what the paper's pseudocode does. It also gives each positive logit the gradient $(\sigma(s_{ii}) - 1)/B$ . That is the same size as the gradient $(A_{ii} - 1)/B$ of CLIP's image-to-caption loss, where $A_{ii}$ is the softmax probability of the right caption. Dividing by $B^2$ would shrink every gradient by another factor of $B$ . Under plain gradient descent the effective learning rate would then fall as the batch grows. Adam divides each gradient by its own running size and so mostly hides a constant factor. But the factor still changes what a gradient-clipping threshold means, how much $\epsilon$ matters, and the weight of this loss against any other loss added to it.

Why a bias. At a batch of 32,768, each positive faces 32,767 negatives. A classifier that has learned nothing should already say "no" to almost every pair. That needs a large negative logit before any cosine has been learned. The scale cannot supply it, because for random embeddings the scaled cosine $e^{\alpha}\hat{\mathbf u}^\top\hat{\mathbf v}$ is centered at zero whatever $\alpha$ is. The bias $b_0$ supplies that prior. A later part of this section shows why $-10$ is the right size.

Why one matrix is enough. The label matrix is symmetric. Swapping images and texts transposes the logit matrix $\mathbf S$ and leaves the loss unchanged. So there is no separate image-to-text and text-to-image loss to average. There is also no pass over each row to subtract its maximum, which a softmax needs to avoid overflow. Every term is computed from its own pair alone.

You have seen this loss before, on words. §1.5 trained word vectors by negative sampling. Each real (word, neighbor) pair was contrasted with $k$ randomly drawn fake neighbors, and each pair was judged by its own sigmoid. The sigmoid loss is the same idea laid out over a batch. Its fakes cost nothing to draw, since they are the other captions already in the batch, and each image gets $B - 1$ of them.

A two-by-two example by hand

Two images and two captions are enough to see every piece of the loss. We reuse the cosines of the InfoNCE hand example in §7.2, so you can compare the numbers with the softmax's, and we fix the scale $e^{\alpha}$ at 10.

✏️ SigLIP on two pairs, with and without the bias

Take $\hat{\mathbf U}\hat{\mathbf V}^\top = \begin{pmatrix}0.8&0.2\\0.4&0.6\end{pmatrix}$ , $e^{\alpha} = 10$ , and $b_0 = -5$ . The logits are

\mathbf S = \begin{pmatrix}8 - 5 & 2 - 5\\ 4 - 5 & 6 - 5\end{pmatrix} = \begin{pmatrix}3 & -3\\ -1 & 1\end{pmatrix}.

The positive $(1,1)$ costs $\log(1 + e^{-3}) = 0.0486$ . So does the negative $(1,2)$ , since $z_{12}s_{12} = 3$ as well. In row 2, $z_{21}s_{21} = z_{22}s_{22} = 1$ , so both entries cost $\log(1 + e^{-1}) = 0.3133$ . The loss is $(2 \times 0.0486 + 2\times 0.3133)/2 = 0.3618$ . The logit gradients $(\sigma(s_{ij}) - \mathbb 1[i=j])/B$ are

\frac{\partial\mathcal L}{\partial\mathbf S} = \begin{pmatrix}-0.0237 & 0.0237\\ 0.1345 & -0.1345\end{pmatrix}.

Image 2 gets about six times the gradient of image 1. Both of its logits sit only 1 away from the decision boundary at 0, against 3 for image 1. The bias gradient is the sum of all entries, and here it is exactly 0. In each row the positive and the negative sit symmetrically about $-b_0 = 5$ , with scaled cosines 8 and 2 in row 1 and 4 and 6 in row 2.

Now set $b_0 = 0$ . The logits are $\begin{pmatrix}8 & 2\\4 & 6\end{pmatrix}$ . The negatives at 2 and 4 cost $\log(1 + e^{2}) = 2.127$ and $\log(1 + e^{4}) = 4.018$ , and the loss jumps to $3.074$ . The bias gradient is $\partial\mathcal L/\partial b_0 = 0.930$ , so gradient descent drags $b_0$ down. The gradient with respect to the log scale is $\partial\mathcal L/\partial\alpha = +2.836$ , so descent also shrinks $e^{\alpha}$ . Without a bias, the only way to lower the negatives' logits is to shrink the scale, and that weakens the positives too.

Python · cell 1

The cell also prints the log-scale gradient $\partial\mathcal L/\partial\alpha$ at $b_0 = -5$ , which is $-0.411$ . A descent step therefore raises $e^{\alpha}$ and makes every answer more confident. §30.2 found the same pressure toward a large scale for CLIP's learned temperature, which is why CLIP clips it at 100.

The gradient, pair by pair

The loss is a sum of separate logistic losses. So its gradient is the familiar logistic gradient, "prediction minus label" (§0.2), one entry at a time.

∴ Gradient of the SigLIP loss

With $\mathbf S$ as above, write $y_{ij} = \mathbb 1[i=j] = (1 + z_{ij})/2$ and $\mathbf G = \partial\mathcal L_{\text{sig}}/\partial\mathbf S \in \R^{B\times B}$ . Then

G_{ij} = \frac{\sigma(s_{ij}) - y_{ij}}{B}, \qquad \nabla_{\hat{\mathbf U}}\mathcal L = e^{\alpha}\,\mathbf G\hat{\mathbf V}, \qquad \nabla_{\hat{\mathbf V}}\mathcal L = e^{\alpha}\,\mathbf G^\top\hat{\mathbf U},

\frac{\partial\mathcal L}{\partial\alpha} = \sum_{i,j} G_{ij}\,(s_{ij} - b_0), \qquad \frac{\partial\mathcal L}{\partial b_0} = \sum_{i,j} G_{ij}.

The derivative of the step that divides each embedding by its length, worked out in §30.2, then carries $\nabla_{\hat{\mathbf U}}$ and $\nabla_{\hat{\mathbf V}}$ back to the raw embeddings.

∂ Derive it, one logistic term at a time

For one term $\ell(s, z) = \log(1 + e^{-zs})$ , the chain rule gives $\partial\ell/\partial s = -z\,e^{-zs}/(1 + e^{-zs}) = -z\,\sigma(-zs)$ . If $z = +1$ this is $-\sigma(-s) = \sigma(s) - 1$ . If $z = -1$ it is $\sigma(s)$ . Both cases are $\sigma(s) - y$ with $y = (1 + z)/2$ . Each $s_{ij}$ appears in exactly one term of the loss, and the loss carries a factor $1/B$ , so $G_{ij} = (\sigma(s_{ij}) - y_{ij})/B$ .

Next, $s_{ij} = e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j + b_0$ gives $\partial s_{ij}/\partial\hat{\mathbf u}_i = e^{\alpha}\hat{\mathbf v}_j$ and $\partial s_{ij}/\partial\hat{\mathbf v}_j = e^{\alpha}\hat{\mathbf u}_i$ . The vector $\hat{\mathbf u}_i$ appears in the $B$ logits of row $i$ , so its gradient is $e^{\alpha}\sum_j G_{ij}\hat{\mathbf v}_j$ , which is row $i$ of $e^{\alpha}\mathbf G\hat{\mathbf V}$ . The vector $\hat{\mathbf v}_j$ appears in column $j$ , which gives row $j$ of $e^{\alpha}\mathbf G^\top\hat{\mathbf U}$ .

Finally, $\partial s_{ij}/\partial\alpha = e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j = s_{ij} - b_0$ , not $s_{ij}$ itself, because the bias is not scaled. And $\partial s_{ij}/\partial b_0 = 1$ for every entry. Summing $G_{ij}$ times these derivatives over all entries gives the last two formulas.

In plain words, negative $j$ pushes image $i$ away with weight $\sigma(s_{ij})$ . That weight is the model's current belief that the pair matches, and it depends on nothing else. The positive pulls with weight $1 - \sigma(s_{ii})$ . A negative whose logit sits well below zero gets almost no gradient, and its cosine is then well below $-b_0/e^{\alpha}$ . So the bias sets one fixed bar for what counts as a hard negative, a wrong caption that the model still finds plausible.

Now compare the softmax. In CLIP's row loss, negative $j$ pushes image $i$ with weight $A_{ij}$ , its probability in the row's softmax. That weight depends on every logit in the row. One identity makes the comparison exact.

∴ A softmax weight is a sigmoid with a moving threshold

For a row of logits $s_{i1},\dots,s_{iB}$ with row softmax $A_{ij} = e^{s_{ij}}/\sum_k e^{s_{ik}}$ , let $\lambda_{ij} = \log\sum_{k\ne j}e^{s_{ik}}$ be the log-sum-exp of the competitors of entry $j$ . Then

A_{ij} = \sigma\big(s_{ij} - \lambda_{ij}\big).

Three consequences follow. (i) The softmax weight of negative $j$ changes when any other logit $s_{ik}$ changes, $\partial A_{ij}/\partial s_{ik} = -A_{ij}A_{ik}$ for $k \ne j$ , while the sigmoid weight $\sigma(s_{ij})$ does not. (ii) The softmax push on a row's negatives sums to $1 - A_{ii} \le 1$ , while the sigmoid push $\sum_{j\ne i}\sigma(s_{ij})$ can be as large as $B - 1$ . (iii) Adding the same constant to every logit of a row leaves every $A_{ij}$ unchanged but moves every $\sigma(s_{ij})$ .

∂ Proof of the identity and its consequences

Divide the numerator and the denominator of $A_{ij}$ by $e^{s_{ij}}$ . That gives $A_{ij} = 1/(1 + e^{\lambda_{ij} - s_{ij}}) = \sigma(s_{ij} - \lambda_{ij})$ .

For (i), the softmax Jacobian from §4.3 is $\partial A_{ij}/\partial s_{ik} = A_{ij}(\mathbb 1[j=k] - A_{ik})$ . For $k \ne j$ that is $-A_{ij}A_{ik} \ne 0$ , while $\sigma(s_{ij})$ is a function of $s_{ij}$ alone. For (ii), each row of $\mathbf A$ sums to 1, while each $\sigma(s_{ij})$ can approach 1 on its own. For (iii), adding $c$ to every logit adds $c$ to both $s_{ij}$ and $\lambda_{ij}$ , so their difference does not change. $\square$

In plain words, both losses compare each negative with a threshold. For the softmax, the threshold of entry $j$ is the log-sum-exp of everything else in its row. That is a statistic of the whole batch, and it moves whenever any pair moves. For the sigmoid, the threshold is $-b_0$ , one learned number shared by every pair. Both thresholds are measured on the scaled cosines $e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j$ , and a bias added inside a softmax would cancel, which is why CLIP has none. That is the whole difference, and it is why the sigmoid never needs to see the whole batch at once.

The cell below makes this concrete for one image. Its positive has logit 6, its six negatives run from 4 down to $-6$ , and the sigmoid uses a bias of $-5$ .

Python · cell 2

Raising negative 1 from 4 to 7 drops the softmax weight on negative 2 from 0.016 to 0.005. The hard negative now takes 0.727 of the row's total push of 0.733, and that total can never exceed 1. The sigmoid weight on negative 2 stays at 0.047. Shifting every logit up by 3 changes nothing for the softmax. The sigmoid's total push grows almost fivefold, from 0.32 to 1.52.

In practice, this means a model trained with the sigmoid is punished for scoring all captions too high, even when it ranks them correctly. A model trained with the softmax is not. An interviewer who asks how the two losses differ wants this threshold picture.

Interview

SigLIP loss and gradients

Implement siglip_loss_grad(Un, Vn, alpha, bias), the forward and the backward pass of the sigmoid loss defined above.

Un and Vn are (B, d) arrays of image and text embeddings. Their rows already have length one, so use them as given. Their gradients are taken with respect to these arrays as plain inputs, without differentiating through any normalization.
alpha is the logarithm of the logit scale, and bias is $b_0$ .
Return (loss, dUn, dVn, dalpha, dbias). That is the loss, with its $1/B$ , followed by its gradients with respect to the four inputs.

Compute $-\log\sigma(t)$ in a form that cannot overflow, and use no Python loops.

import numpy as np

def siglip_loss_grad(Un, Vn, alpha, bias):
    """Un, Vn: (B, d) unit rows. alpha: log of the logit scale. bias: the logit bias b0.
    Returns (loss, dUn, dVn, dalpha, dbias)."""
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Core

Softmax and sigmoid weights on the negatives

For one image's row of logits, compute how hard each negative is pushed under each loss, and check the identity that links the two.

negative_weights(s, pos, bias, mode) takes s, a (B,) array of scaled cosines $e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j$ without the bias, and pos, the index of the positive. It returns a (B,) array. Each entry is the derivative of that row's loss with respect to one negative logit, and the entry at pos is 0.
With mode='softmax' the row's loss is $-\log\operatorname{softmax}(\mathbf s)_{\text{pos}}$ , and bias is ignored. With mode='sigmoid' it is the row's part of the sigmoid loss with bias bias, without the $1/B$ .
softmax_as_sigmoid_bias(s, j) returns the number $\beta_j$ for which $\sigma(s_j + \beta_j)$ equals the softmax weight of entry $j$ .

Both functions must stay finite for logits in the thousands.

import numpy as np

def negative_weights(s, pos, bias, mode):
    """s: (B,) one image's logits e^alpha * cosine, without the bias. pos: index of its positive.
    Returns a (B,) array, the derivative of that row's loss with respect to each negative logit,
    and 0 at pos. mode 'softmax' uses the row softmax (bias ignored), 'sigmoid' the SigLIP terms."""
    w = np.full(len(s), 1.0 / len(s))
    w[pos] = 0.0

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Why the bias starts near minus log B

The starting value of the bias decides how the first steps of training go, and the best value has an exact formula. At the start of training the two towers are random and know nothing about each other. Two random unit vectors in many dimensions are nearly perpendicular (§1.3). So every cosine $\hat{\mathbf u}_i^\top\hat{\mathbf v}_j$ is close to 0, and every logit is close to $b_0$ . Each row then holds one positive and $B - 1$ negatives that all look the same. We want the $b_0$ that makes this starting point least wrong.

The best bias for four pairs made each pair's probability equal to the share of pairs that match, one in four. The same holds at every batch size, and the theorem below proves it by writing one image's starting loss as a function of $b_0$ alone.

∂ Proof of the optimal initial bias

Differentiate term by term, using $\frac{d}{dt}\log(1 + e^{t}) = \sigma(t)$ ,

f'(b_0) = -\sigma(-b_0) + (B - 1)\,\sigma(b_0) = (B-1)\,\sigma(b_0) - \big(1 - \sigma(b_0)\big).

The second derivative is $f''(b_0) = \sigma(b_0)\sigma(-b_0) + (B-1)\sigma(b_0)\sigma(-b_0) = B\,\sigma(b_0)\big(1 - \sigma(b_0)\big) > 0$ . So $f$ is strictly convex, and its stationary point is its minimum.

Setting $f'(b_0) = 0$ gives $\sigma(b_0)/(1 - \sigma(b_0)) = e^{b_0} = 1/(B - 1)$ . So $b_0^\star = -\log(B - 1)$ and $\sigma(b_0^\star) = 1/B$ .

Substituting, $\log(1 + e^{-b_0^\star}) = \log B$ and $(B-1)\log(1 + e^{b_0^\star}) = (B-1)\log\frac{B}{B-1}$ . Since $\log(1 + x) = x - x^2/2 + \dots$ , the second term is $(B-1)\log\big(1 + \frac{1}{B-1}\big) = 1 - \frac{1}{2(B-1)} + \dots$ , which gives the approximation. $\square$

In plain words, the best guess before learning anything is that each pair matches with probability $1/B$ , the fraction of pairs that do. The derivative has a second reading. Its first term, $(B-1)\sigma(b_0)$ , is the total push on a row's $B - 1$ negatives. Its second term, $1 - \sigma(b_0)$ , is the pull on the positive. So $b_0^\star$ is exactly where the push balances the pull. At initialization the ratio of push to pull is $(B-1)e^{b_0}$ .

Python · cell 3

The minimum moves left by about $\log 2 \approx 0.69$ each time the batch doubles. The curves are also lopsided. Above the optimum the slope $f'(b_0)$ climbs toward $B - 1$ , since every negative is then wrong. Below it the slope only approaches $-1$ , since only the positive is wrong.

SigLIP's fixed $-10$ comes within a tenth of a nat of the best starting loss for batches from about 14,000 to 33,000. (A nat is the unit of a loss measured with natural logarithms.) It comes within half a nat for batches from about 7,000 to 52,000. At smaller batches it sits below the optimum, on the gentle side.

What goes wrong at $b_0 = 0$ is the gradient, not only the loss value. The first updates are dominated by $B - 1$ negatives per row, all saying "lower me". That push carries no information about which caption belongs to which image. The paper describes large early steps that over-correct for the imbalance, and its ablation shows the cost. It trained a Base-size Vision Transformer (86 million parameters, §27.2) at a batch of 8,192 for 900 million examples, with the scale starting at 10. Zero-shot ImageNet accuracy was 62.0 percent with no bias, 61.7 with $b_0 = 0$ , and 63.0 with $b_0 = -10$ . Starting the scale at 1 instead hurt the run with $b_0 = 0$ far more (53.7 percent) than the run with $b_0 = -10$ (61.0).

The same effect shows up in a small training run that you can execute here. A hidden "concept" of 8 numbers is seen through two noisy views of 24 features each, which stand in for an image and its caption. Two linear towers map each view into a shared space of width 64. The loop trains them with Adam and learns the scale and the bias too. It sets $\beta_2 = 0.95$ , the decay rate of Adam's running average of squared gradients, for a reason given later. The gradients are those of the theorem above, carried back through the normalization.

The same code also trains CLIP's softmax loss, which we use later to compare the two. We score each run by recall@1 on 500 held-out pairs. That is the fraction of images whose top-scoring caption is their own, averaged with the same fraction for captions.

Python · cell 4

Python · cell 5

Starting at $b_0 = 0$ wrecks the run, with recall@1 below 1 percent after 40 steps. Starting at $-3$ reaches only 0.23, because the first updates spend themselves pushing every pair apart. The optimal value reaches 0.72. And $-10$ , which starts below the optimum, reaches 0.81.

Two facts explain why erring low is safe. First, when the push of the negatives is weaker than the pull of the positive, the first steps pull matched pairs together. That is exactly the information the model needs, and the bias then rises on its own as the negatives start to matter. Second, the theorem assumed that every cosine is exactly 0. Here, random 64-dimensional embeddings give cosines with a standard deviation of about $1/\sqrt{64}$ (§1.3). So the logits spread around $b_0$ by about $e^{\alpha}/\sqrt{64} = 1.25$ . That spread moves the best starting bias down, to about $-6.3$ here, as the box below derives.

In practice, start at $-10$ for batches in the tens of thousands, or at $-\log(B-1)$ for other sizes, and lean lower rather than higher. An interviewer will ask you to derive $-\log(B-1)$ and to say why 0 is a bad start.

🔬 The initial bias when the logits are spread

Model the starting logits as $s = b_0 + X$ with $X \sim \mathcal N(0, v)$ , a Gaussian of variance $v$ . The positive and the negatives get the same distribution, since at initialization the towers cannot tell them apart. Differentiate the expected loss per image inside the expectation, exactly as in the proof above,

\frac{d}{db_0}\,\mathbb E\big[\mathcal L\big] = (B-1)\,\mathbb E\big[\sigma(b_0 + X)\big] - \mathbb E\big[1 - \sigma(b_0 + X)\big] = B\,\mathbb E\big[\sigma(b_0 + X)\big] - 1 .

So the stationary bias solves $\mathbb E[\sigma(b_0 + X)] = 1/B$ . The left side increases with $b_0$ , so the root is unique. With $v = 0$ the condition is $\sigma(b_0) = 1/B$ , the theorem again.

For $v > 0$ there is no closed form, but there is a good approximation. Far below zero, $\sigma(t) \approx e^{t}$ . The mean of $e^{X}$ is $e^{v/2}$ , which follows from completing the square in the Gaussian integral, $\int e^{x}e^{-x^2/2v}dx = e^{v/2}\int e^{-(x - v)^2/2v}dx$ . So the condition becomes $e^{b_0 + v/2} \approx 1/B$ , where without spread it was $e^{b_0} \approx 1/B$ . The bias moves down by about half the logit variance, to roughly $-\log(B-1) - v/2$ . In the run above that is $-5.54 - 0.78 = -6.32$ , and the exact root is $-6.31$ . At a real joint width of 768 the shift is only $0.07$ .

The direction of the shift is Jensen's inequality. In this region $\sigma$ is convex, so $\mathbb E[\sigma(b_0 + X)] > \sigma(b_0)$ , and the spread raises the average push of the negatives. Putting the mean logit into $\sigma$ , instead of averaging $\sigma$ over the logits, misses the effect entirely.

Core

The right initial bias, with and without spread

Implement two functions that give the starting bias for a batch of B pairs. The second one uses the condition derived in the box above.

init_bias(B) returns the bias that minimizes the loss per image when every logit equals b0.
init_bias_spread(B, logit_std) returns the bias at which the expected starting loss per image is stationary, when every logit, positive or negative, is b0 plus independent Gaussian noise with standard deviation logit_std.
Compute the expectation with Gauss-Hermite quadrature, np.polynomial.hermite_e.hermegauss. It gives a fixed set of points and weights for integrating against a Gaussian. Find the root with a bracketed one-dimensional search such as bisection.

The second function must reduce to init_bias(B) at logit_std = 0. It must stay finite and accurate for B up to 32,768 and logit_std up to 3. Do not use the approximation $-\log(B-1) - v/2$ , which fails at small B and large spread.

import numpy as np

def init_bias(B):
    """The bias that minimizes the initial loss per image when every logit equals it."""
    return -10.0

def init_bias_spread(B, logit_std):
    """The bias b0 at which the expected initial loss per image is stationary when every logit is

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

What a perfectly trained score measures

The bias result is a special case of a cleaner fact. That fact says what a perfectly trained sigmoid score measures. The answer is how much more often an image and a text occur together than they would by chance.

That quantity is the pointwise mutual information, $\operatorname{PMI}(x, y) = \log\frac{p(x,y)}{p(x)p(y)}$ , which §1.4 met for words. Here is an example. Suppose a photo of a beach and the caption "waves at sunset" appear together four times as often as independent draws would pair them. Their PMI is $\log 4 = 1.39$ . If they appear together exactly as often as chance predicts, their PMI is 0.

To state the result, let the pairs of a batch be drawn independently from a joint distribution $p(x, y)$ of images and texts, with marginals $p(x)$ and $p(y)$ . Let $s(x, y)$ be any scoring function, not only a scaled cosine plus a bias.

∂ Proof that the optimum is PMI minus log(B - 1)

Row $i$ contributes $\log(1 + e^{-s(x_i, y_i)}) + \sum_{j\ne i}\log(1 + e^{s(x_i, y_j)})$ . The pair $(x_i, y_i)$ is drawn from $p(x, y)$ . For $j \ne i$ , $y_j$ belongs to a different, independently drawn pair, so $(x_i, y_j)$ is distributed as $p(x)p(y)$ . By linearity every row has the same expected value, the right-hand side above, and the $\frac1B\sum_i$ averages $B$ copies of it. Unlike the softmax loss's expectation, this one is exact at every $B$ , because no normalizer couples the terms.

For discrete images and texts the expectation is $\sum_{x,y}\big[p(x,y)\log(1 + e^{-s}) + (B-1)p(x)p(y)\log(1 + e^{s})\big]$ . Each $s(x, y)$ appears in one term only, so we can minimize term by term. The derivative of the $(x, y)$ term is $-p(x,y)\sigma(-s) + (B-1)p(x)p(y)\sigma(s)$ . It increases with $s$ and vanishes when $e^{s} = \frac{p(x,y)}{(B-1)p(x)p(y)}$ , that is, at $s = \operatorname{PMI}(x,y) - \log(B-1)$ . If $p(x, y) = 0$ but $p(x)p(y) > 0$ , the derivative is positive everywhere, and the optimal score runs to $-\infty$ . $\square$

In plain words, a perfectly trained sigmoid logit is the PMI of the image and the text, shifted down by $\log(B-1)$ . That shift is the log-odds of the prior that a random pair in the batch matches, one chance in $B$ . (The log-odds of a probability $p$ is $\log\frac{p}{1-p}$ , and for $p = 1/B$ it is $-\log(B-1)$ .) Write the logit as $s = e^{\alpha}\hat{\mathbf u}^\top\hat{\mathbf v} + b_0$ with $b_0 = -\log(B-1)$ . Then the scaled cosine is the PMI itself. At initialization the towers know nothing, so the model acts as if PMI were 0 everywhere, and the theorem gives back $b_0^\star = -\log(B-1)$ .

You have met this result twice before. Negative sampling with $k$ fake neighbors per real one is optimized by $\operatorname{PMI} - \log k$ (§1.5), and here $k = B - 1$ . CLIP's softmax loss is optimized by PMI plus a constant (§30.1). But the softmax never sees that constant, because adding one number to every logit leaves every softmax unchanged. So its value is arbitrary. The sigmoid pins it at $-\log(B-1)$ .

As a result, a sigmoid logit has an absolute meaning, and one threshold works across all queries. A CLIP logit can be compared with other logits, but no value of it means "match" on its own. That is the difference an interviewer is probing when they ask whether a CLIP score is a probability.

Python · cell 6

The cell builds a random joint distribution over six images and six texts. It then minimizes the expected loss numerically, with one free score per (image, text) cell. The minimizer differs from PMI by exactly $-\log(B-1)$ in every cell, at every batch size.

What the sigmoid of a trained score means

It is tempting to read $\sigma(s)$ as the probability that the pair matches. That is almost right, and the gap is a favorite follow-up question. Substitute the optimum into the sigmoid. Since $e^{s^\star} = \frac{p(x,y)}{(B-1)p(x)p(y)}$ ,

\sigma\big(s^\star(x,y)\big) = \frac{1}{1 + e^{-s^\star}} = \frac{p(x,y)}{p(x,y) + (B-1)\,p(x)\,p(y)} = \frac{\tfrac1B\,p(x,y)}{\tfrac1B\,p(x,y) + \tfrac{B-1}{B}\,p(x)\,p(y)} .

The last form is Bayes' rule for one specific question. Draw a pair the way a batch does. With probability $1/B$ it is a positive, drawn from the joint $p(x, y)$ . With probability $(B-1)/B$ it is a negative, an image and a text drawn independently from $p(x)p(y)$ . Then $\sigma(s^\star)$ is the posterior probability that the pair was a positive. So it is a match probability under the training prior $1/B$ , and that prior is a property of the batch, not of the world.

∂ Proof of the re-biasing rule

By Bayes' rule the posterior is $\pi p(x,y)/\big(\pi p(x,y) + (1-\pi)p(x)p(y)\big)$ . Divide the numerator and the denominator by $\pi p(x,y)$ . That gives $1/(1 + e^{-s'})$ with $e^{-s'} = \frac{1-\pi}{\pi}\cdot\frac{p(x)p(y)}{p(x,y)}$ . So $s' = \operatorname{PMI} + \log\frac{\pi}{1-\pi}$ , and $\operatorname{PMI} = s^\star + \log(B-1)$ by the theorem above. $\square$

In plain words, to use a sigmoid score as a probability in a new setting, add back the $\log(B-1)$ that training subtracted, then add the log-odds of the new base rate. Only the bias changes, so one threshold still serves every query once it is re-set. With $\pi = 1/B$ the correction vanishes, as it should.

Three cautions keep this honest.

A trained model is not at the population optimum. Its logits approximate $s^\star$ only roughly, and the learned bias drifts away from $-\log(B-1)$ during training (the paper saw it move when negatives were removed). So fit or check any threshold on held-out labeled pairs, as with the calibration of §30.3.
The derivation assumed that non-matches at deployment look like random pairs. A shortlist from a first-stage retriever is full of near misses, whose PMI is far above that of a random pair, so the same score means less there.
Shift the logit, not the probability, because Bayes' rule adds in log-odds.

Python · cell 7

The same raw score can mean near certainty or a long shot. That is why a production system sets its threshold for the base rate it actually faces, and why "is $\sigma(s)$ a probability?" is a good interview question.

What decoupling means, and what it does not

Now we can say precisely what the paper means when it says that the sigmoid loss "decouples the batch size from the definition of the task".

📌 What decoupling means, and what it does not

The loss is decoupled from the batch size in two senses.

Each term of the loss depends on one pair only. So a pair's loss does not change when other pairs join the batch, and any block of the matrix can be computed on its own, with no statistic of the whole batch.
The optimal score is PMI minus a constant that depends only on $B$ , and one learned number, the bias, absorbs that constant. So the target for the embeddings, $e^{\alpha}\hat{\mathbf u}^\top\hat{\mathbf v} = \operatorname{PMI}$ , is the same at every batch size.

It is not decoupled in three other senses.

The prior of a match is $1/B$ . So the right bias, the starting loss (about $\log B + 1$ per image), and the balance of positive and negative gradient all move with $B$ .
A finite model cannot hit PMI exactly. In its approximation the factor $B - 1$ weights the errors on negatives against those on positives, much as the word counts weight the factorization that negative sampling learns for word vectors (§1.5).
The number of negatives per image grows with $B$ , and with it the chance of meeting a hard one.

What the batch size does, measured

How large a batch does each loss need? The paper answers in two settings, both with Vision Transformers. Their names give a size and a patch width (§27.2). B/16 is the Base size, 86 million parameters, cutting images into 16-pixel patches. g/14 is the giant size, about a billion parameters, with 14-pixel patches.

The first setting freezes a pretrained giant image tower, stores its image embeddings once, and trains only a Base-size text tower from scratch. This is locked-image tuning, which a later part of this section covers, and with the sigmoid loss the paper calls it SigLiT. Its batches ran from 512 to about one million, for 18 billion examples seen. The second setting trains both towers, at Base size, from scratch for 9 billion examples, at batches up to 307,000.

Our small run can show the small-batch half of that result. The cell trains both losses on the same synthetic pairs, with the same number of pairs seen, once with a batch of 16 and once with 256.

Python · cell 8

At a batch of 16 the sigmoid wins clearly, 0.78 against 0.71. At 256 the softmax is slightly ahead in this toy, 0.82 against 0.81. One plausible reading of the small-batch gap is that a softmax over 16 candidates is a noisy normalizer. Its gradient on each negative depends on which 15 others happened to be drawn, while the sigmoid's gradient does not. Do not read more into the large-batch numbers than the paper's own result, which is that the two losses converge. In practice, if your hardware caps you at a few thousand pairs per step, the sigmoid is the safer choice.

Three more findings from the paper complete the picture.

The imbalance is not the problem it looks like. At a batch of 16,384 there are about 268 million negative pairs for 16,384 positives. In the frozen-image setting, the paper masked out negatives to rebalance the batch, down to 1.6 negatives per positive, and it did not help. Dropping negatives at random hurt. Keeping only the easiest ones failed completely. Keeping only the hardest ones nearly matched the full batch, and slightly beat it when trained longer to see as many pairs. So most of what the negatives teach comes from the hard ones. The learned bias also rose as negatives were removed, from about $-12.5$ to about $-3.5$ , absorbing the changed prior as the theorem says it should.

Robustness to noisy data. The paper corrupted a rising fraction of the training data. It replaced images with noise, replaced texts with random tokens, or shuffled which caption went with which image. Models trained with the sigmoid kept their lead over models trained with the softmax at every level of corruption. Earlier work had found the same for sigmoid losses in classification. Web alt-text is noisy, so this matters.

Spikes at large batches. Larger batches made training less stable. Spikes in the gradient norm caused large jumps in the parameters. Lowering $\beta_2$ from 0.999 to 0.95 fixed it, in both Adam and Adafactor (a memory-saving variant of Adam). $\beta_2$ sets how long the optimizer's running average of squared gradients remembers. At 0.95 it averages over about 20 steps instead of 1,000, so it recovers from a spike quickly. Language model pretraining uses the same remedy, for the same reason (§8.6, §8.8).

Computing the loss on a ring of devices

A large batch is split across many accelerators. The loss must then be computed without any one of them holding the whole batch. Let $D = n_{\text{dp}}$ be the number of data-parallel devices, each holding $b = B/D$ pairs, its shard of the batch.

Recall how CLIP does it (§30.2). An all-gather, a collective operation that gives every device a copy of every device's data, puts all $B$ image and text embeddings on every device. Each device then computes its $b\times B$ rows and $B\times b$ columns of logits.

The sigmoid loss needs neither the gather nor the slices. Every term depends on one pair, so the $B\times B$ matrix can be computed one $b\times b$ block at a time, in any order.

Picture four devices around a table, each holding its own images and its own captions. First, each device scores its images against its own captions. That is the only block with positives. Then every device passes its captions to its neighbor around the table, all in the same direction. It scores its images against the captions it just received and adds that all-negative block to its running total. After three passes every device has seen every caption once, and nobody ever held more than one block. The widget runs this schedule.

That is the chunked implementation of Zhai et al. (2023). Device $r$ keeps its own images $\hat{\mathbf U}_r$ and starts with its own texts $\hat{\mathbf V}_r$ . At step 0 it scores the block $(r, r)$ . Then every device sends the text shard it holds to device $r - 1$ . This step is a collective permute, in which every device sends to one fixed partner and receives from another at the same time. So device $r$ receives what device $r + 1$ held, scores the new block, and adds it to its sum. After $D - 1$ permutes, a final sum across devices and a division by $B$ give the loss.

∴ The ring computes the exact loss

Let device $r \in \{0,\dots,D-1\}$ hold text shard $(r + k) \bmod D$ at step $k = 0,\dots,D-1$ . Let it add the summed pair losses of block $(r, (r+k)\bmod D)$ , with positive labels on the block's diagonal only when $k = 0$ . Then every ordered pair $(i, j)$ is scored exactly once, with its correct label, and

\frac{1}{B}\sum_{r=0}^{D-1}\sum_{k=0}^{D-1}\;\sum_{i\in\mathcal I_r}\;\sum_{j\in\mathcal I_{(r+k) \bmod D}}\log\big(1 + e^{-z_{ij}s_{ij}}\big) = \mathcal L_{\text{sig}},

where $\mathcal I_r$ is the set of indices on device $r$ . No device ever holds more than $b^2$ logits.

∂ Proof that the ring visits every pair once

Pair $(i, j)$ lies in block $(r, c)$ with $i \in \mathcal I_r$ and $j \in \mathcal I_c$ . For a fixed $r$ , the map $k \mapsto (r + k) \bmod D$ is a bijection from $\{0,\dots,D-1\}$ onto itself. So block $(r, c)$ is scored at exactly one step, $k = (c - r)\bmod D$ . Positives have $i = j$ , so they lie in blocks with $c = r$ , which are scored at $k = 0$ , and within such a block they are its diagonal. Every other block holds only negatives. The double sum therefore visits each of the $B^2$ terms of $\mathcal L_{\text{sig}}$ exactly once, with the right label. $\square$

The schedule fixes the costs, since each device holds one block at a time and passes one text shard per step.

Memory. One $b\times b$ block of logits at a time, plus a second text buffer if the next permute overlaps the current block's compute. The sharded softmax holds $2bB$ logits.
Communication in the forward pass. $D - 1$ permutes of a $b\times d$ text shard, $(D-1)bd$ numbers per device. That is half of the $2(D-1)bd$ of the two all-gathers, using the ring cost of §10.2. Each permute talks only to a neighbor and can overlap with the block's compute. The paper reports that $D$ collective permutes are typically faster than two all-gathers.
The backward pass. Each block's logit gradient $(\sigma(s_{ij}) - y_{ij})/B$ is known the moment the block is computed, so the backward pass can run inside the same loop. The image gradient $e^{\alpha}\mathbf G_{\text{blk}}\hat{\mathbf V}_c$ stays on its device and accumulates there. The text gradient $e^{\alpha}\mathbf G_{\text{blk}}^\top\hat{\mathbf U}_r$ accumulates in a buffer that travels around the ring with its text shard and returns to the shard's owner.

🔬 Why the softmax does not ring as easily

A ring is possible for the softmax too, but it carries more state. Each row's normalizer and each column's normalizer must be accumulated across all $D$ steps. That uses the running log-sum-exp of §12.2, the trick that lets attention kernels process a row in pieces. The column statistics travel with the text shards. Write $\bar A_{ij}$ for the column softmax, the probability of image $i$ among all images for caption $j$ . No block's gradient $(A_{ij} + \bar A_{ij} - 2\,\mathbb 1[i=j])/(2B)$ can be formed until those normalizers are final, so the backward pass needs a second trip around the ring. The independence of the sigmoid terms is what makes the one-pass ring natural. The paper credits it with enabling batches above one million on relatively few devices.

Python · cell 9

So the ring turns a batch size that is impossible for the softmax into a routine one. In a systems interview, the expected answer to "how would you train a contrastive model at a batch of a million?" is this ring, with its memory and its traffic.

Interview

The chunked ring

Implement ring_siglip_loss(U_shards, V_shards, alpha, bias), a simulation of the ring on one machine.

The inputs are lists of D arrays of shape (b, d), the unit image and text embeddings on each device.
Device r starts with text shard r. After every step, each device receives the shard held by device (r + 1) % D.
Compute every block's summed pair losses with the provided block_loss(U_blk, V_blk, alpha, bias, has_pos). It records the shape of every block it is called with, and no block larger than (b, b) may be formed.
Return (total, visits, schedule). Here total is the SigLIP loss of the whole batch, and visits is a (B, B) integer array counting how many times each (image, text) pair was scored. schedule[k][r] is the index of the text shard that device r holds at step k.

import numpy as np

def ring_siglip_loss(U_shards, V_shards, alpha, bias):
    """U_shards, V_shards: lists of D arrays of shape (b, d), device r's unit image and text embeddings.
    Returns (total, visits, schedule): the SigLIP loss of the whole batch, a (B, B) int array counting
    how often each (image, text) pair was scored, and schedule[k][r], the text shard device r holds at step k."""
    D = len(U_shards)
    total = sum(block_loss(U_shards[r], V_shards[r], alpha, bias, True) for r in range(D))   # local blocks only

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In PyTorch

Production code computes the loss with F.logsigmoid, a log of the sigmoid that never overflows, and lets autograd compute the gradients. The block below checks autograd against the closed-form gradients of the theorem. Then it computes the same loss block by block over three simulated devices.

🔥 In PyTorchfor a machine with PyTorch

On real devices the permute is a point-to-point exchange between processes. It is wrapped in an autograd function whose backward pass sends the gradient the opposite way around the ring. OpenCLIP, the widely used open-source library for training CLIP models, implements its SigLIP loss this way. The code below cannot run here, since it needs several processes.

The last line is worth a second look, because it hides a classic bug. Each rank (one process, driving one device) divides by its own $b$ . Data parallelism then averages the parameter gradients over the $D$ ranks, so the effective normalizer is $bD = B$ , matching the definition. Dividing by $B$ on each rank would make the gradients $D$ times too small.

Locking the image tower

Everything so far trained both towers from scratch. Suppose instead you already own an excellent image model, trained on far more images than you have captions. A cheaper and often better option is to freeze it and train only a text tower that learns to land on its embeddings. Zhai et al. (2022) studied this choice and named the options with two letters, the first for the image tower and the second for the text tower.

Locking has engineering benefits that are easy to see. No gradients flow through the image tower, so training is faster and needs less memory. Without image augmentation, the image embeddings can be computed once for the whole dataset and stored. After that, each step costs only the text tower and the logits.

The SigLIP paper shows what that buys. SigLiT on four TPUv4 chips reached 79.7 percent zero-shot ImageNet accuracy in one day. It used a frozen B/8 image model and a text tower of Large width cut to 12 layers instead of 24. With a frozen g/14 image tower and a Large text tower, it reached 84.5 percent in two days.

The surprising result is that locked also beats unlocked. In the LiT study, Lu almost always beat Uu and uu across training lengths. On a dataset of 4 billion image-text pairs, Lu reached 70.1 percent zero-shot ImageNet, against 57.2 for Uu and 50.6 for uu. A LiT model on a g/14 image tower reached 85.2 percent zero-shot on ImageNet. It reached 82.5 percent on ObjectNet, a test set of objects in unusual poses and backgrounds.

The paper explains this with a measured trade-off. Unlocking the image tower reaches a lower contrastive loss on the caption data it trains on. But it reaches a higher loss on other caption sets. And its features do worse in a few-shot linear probe, a linear classifier trained on a few labeled images per class on top of the frozen features. Contrastive fine-tuning narrows a general representation to one dataset. Locking keeps the general representation and teaches the text side to match it. A pretrained text tower helped only a little, and locking the text tower did not work.

Locking has limits, and each one is a good interview follow-up.

The image tower cannot learn what its pretraining missed. Concepts that are rare in its pretraining data, such as those of specialized domains, stay poorly separated. The text tower can only point at what is already there.
The LiT paper reports that on cross-modal retrieval, unlike classification, Lu's advantage is less clear. With very long schedules, Uu or UU sometimes overtakes it.
If you do unlock a pretrained tower, protect it. The SigLIP paper fine-tuned a public B/16 image checkpoint with its learning rate scaled by 0.1. The default weight decay degraded the pretrained weights. On a 10-shot ImageNet probe the fine-tuned tower ended barely better than one trained from scratch. Turning off weight decay on the pretrained weights, while keeping it on the new text tower, gave 71.0 percent zero-shot on 16 TPUv4 chips in three days.

The idea is alive in current work. dino.txt (Jose et al. 2024) trains a text encoder from scratch against a frozen DINOv2, an image encoder trained without any text (§28.4). It also trains two new transformer blocks on top of the frozen backbone. It concatenates the average of the patch tokens with the class token, so that dense, pixel-level tasks line up with text too. The same recipe can give a language interface to any of the self-supervised image encoders of Chapter 28.

Research

Locked-image tuning on frozen features

The setup builds a small world of eight classes, with only 64 image-caption pairs to train on.

Raw images are a class pattern buried in heavy pixel noise.
Captions are bags of words, the class word plus three filler words.
W_pre is a pretrained image tower that already removes the noise. It stands in for a backbone trained on far more data.

Implement lit_train(Zi, Ht, Wt0, steps, lr, alpha, bias), which trains a linear text tower against frozen image embeddings.

Zi is (B, k), the frozen unit image embeddings unit(Xtr @ W_pre).
Ht is (B, V), the caption features, and Wt0 is (V, k), the initial linear text tower.
Train only Wt, with steps steps of plain full-batch gradient descent, Wt = Wt - lr * dL/dWt. The loss is the SigLIP loss of Zi against normalize(Ht @ Wt), at a fixed scale $e^{\alpha}$ and a fixed bias.
Return Wt, and do not modify Zi or Wt0.

The tests check that your steps match gradient descent exactly and that the loss falls. Then they classify 400 held-out images zero-shot, with the prompt "a photo of a" plus each class word. Your model must beat one whose two towers were both trained from scratch on the same 64 pairs with the same budget.

import numpy as np

def lit_train(Zi, Ht, Wt0, steps, lr, alpha, bias):
    """Zi: (B, k) frozen unit image embeddings. Ht: (B, V) text features. Wt0: (V, k) initial text tower.
    Train only Wt with `steps` steps of full-batch gradient descent, Wt <- Wt - lr * dL/dWt, on the SigLIP
    loss of Zi against normalize(Ht @ Wt) at fixed scale e^alpha and bias. Return Wt."""
    return Wt0

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

With the default seeds, LiT reaches about 89 percent zero-shot accuracy on the held-out images, against about 59 percent for the two towers trained from scratch on the same 64 pairs. Both models share a handicap. With eight classes, a batch of 64 holds about seven other captions of each image's class, and all of them are labeled as negatives. These false negatives, wrong labels on pairs that actually fit, are a fact of contrastive training on web data. §30.5 shows how curation reduces them, and the distillation part of this section shows how soft targets soften them.

Adding a captioning loss

A contrastive loss teaches an encoder only what it needs to tell captions apart within a batch. If no other caption in the batch mentions a red hat, the encoder never has to notice the hat. A captioning loss asks for more. To predict every word of the caption, the image features must carry everything the words mention. For an image $x$ and caption tokens $y_1,\dots,y_T$ , a decoder that reads the image features is trained on

\mathcal L_{\text{cap}} = -\sum_{t=1}^{T}\log p_\theta\big(y_t \mid y_{<t}, x\big),

the next-token loss of §3.1, conditioned on the image.

CoCa (Yu et al. 2022) adds this loss to a contrastive one in a single forward pass. Its text decoder is split in half.

The bottom half has causal self-attention only and never sees the image. Its output at a learned token appended to the end of the caption is a pure text embedding for the contrastive loss.
The top half adds cross-attention to the image encoder's output tokens and predicts the caption.

The total loss is $\mathcal L_{\text{CoCa}} = \lambda_{\text{con}}\mathcal L_{\text{con}} + \lambda_{\text{cap}}\mathcal L_{\text{cap}}$ with $\lambda_{\text{cap}} = 2$ and $\lambda_{\text{con}} = 1$ , since a larger captioning weight was better in their ablation. The image side is pooled by attention, with 1 learned query for the contrastive embedding and 256 for the decoder.

CoCa trained with a softmax contrastive loss at a batch of 65,536, for about 5 days on 2,048 TPUv4 chips. Its data were Google's internal JFT images, with their class labels used as text, plus the web alt-text collected for ALIGN, an earlier model trained like CLIP (§30.2). It reached 86.3 percent zero-shot on ImageNet, 90.6 percent with a frozen encoder and a learned classification head, and 91.0 percent fine-tuned.

Can captioning stand alone? Tschannen et al. (2023) compared captioning and contrastive pretraining with matched data, compute, and model size. Captioning alone gave vision encoders that were competitive with contrastive ones on classification, better on vision-language tasks, and scaled at least as well. Their variant CapPa predicts the whole caption at once from mask tokens, without the causal mask, for 75 percent of training examples. That forces the decoder to rely on the image rather than on the previous words.

The price is zero-shot classification. A captioner must score every class name with its decoder for every image. A contrastive model compares one stored embedding per class.

The practical lesson is that a captioning loss buys richer patch features at the cost of cheap zero-shot classification. That trade decides whether you want a contrastive, a generative, or a combined encoder. The combined one is the next design.

SigLIP 2 combines the pieces

SigLIP 2 (Tschannen et al. 2025) keeps SigLIP's architecture, so its weights drop into existing systems. On top of the sigmoid loss it stacks several losses and data techniques that were developed separately. Here is the recipe, with each piece said in plain words and credited.

📌 The SigLIP 2 recipe

Losses. From the start, training adds to the sigmoid loss, with equal weight, a decoder that reads the unpooled patch tokens and learns where things are, following LocCa (Wan et al. 2024). The decoder has three targets. It writes captions, predicts the box for a caption of a region, and predicts the caption for a given box. Half the examples use the parallel caption prediction of CapPa. The decoder is used only in training and is not released.

For the last 20 percent of training, two self-supervised losses are added, following SILC (Naeem et al. 2023) and TIPS (Maninis et al. 2024). Both use a teacher whose weights are a running average of the model's own. One loss matches the model's features of 8 small crops to the teacher's features of the full image. The other masks half of the patches and asks the model to predict the teacher's features for them.

Data and text. WebLI, Google's web collection of 10 billion images and 12 billion alt-texts in 109 languages, mixed 90 percent English and 10 percent other languages. Filters reduce biases in how groups are represented and what they are associated with. Text is lower-cased and cut to 64 tokens of the multilingual Gemma tokenizer, which has 256,000 tokens.

Training. A batch of 32,768, 40 billion examples, and Adam with learning rate $10^{-3}$ , decoupled weight decay $10^{-4}$ , and gradient clipping at norm 1. Up to 2,048 TPUv5e chips, with fully sharded data parallelism, which splits the parameters and optimizer state across devices. Both towers pool their tokens with a small attention layer that has one learned query (§27.2).

Variants. Sizes B (86M parameters), L (303M, as the paper counts its L/16), So400m (400M, a ViT whose width and depth were tuned for its compute), and g (1B). NaFlex checkpoints keep each image's native aspect ratio and a variable number of patches. They resize the learned position embeddings to fit and mask the padding. This combines the native-aspect inputs of NaViT (Dehghani et al. 2023), which §31.5 explains, with the several sequence lengths that one FlexiViT model serves (Beyer et al. 2023). The two smallest models, B/16 and B/32, are fine-tuned for 4 billion more examples with active data curation (Udandarao et al. 2024). A teacher and the current model score how much each candidate example would teach. The best 32,768 candidates form the batch, picked from 65,536 for B/16 and from 131,072 for B/32, so the teacher's knowledge passes on through the choice of data.

Zero-shot ImageNet top-1 (Table 1 of Tschannen et al. 2025)	SigLIP	SigLIP 2
B/16 at 224 px	76.2	78.2
L/16 at 256 px	80.5	82.5
So400m/14 at 384 px	83.2	84.1
g/16 at 384 px	(none)	85.0

The gains on zero-shot ImageNet are 1 to 2 points at each size. Why does a captioning decoder help an encoder that will be used without it? The decoder's gradient reaches the patch tokens, not just the pooled embedding. And it asks them to support text about specific regions and boxes. The contrastive loss alone shapes the pooled vector only enough to rank captions within a batch.

So the paper's largest gains are on localization and dense prediction, such as finding the region a phrase describes, segmentation, and depth. These are the weaknesses that §30.6 examines. Inside a vision-language model (VLM), SigLIP 2 also beats SigLIP as the image encoder. In practice, if you need an off-the-shelf image tower for a VLM or a dense task, SigLIP 2 is a strong default.

Distilling a joint encoder

Everything above makes encoders better. Serving often needs them smaller. A phone app that embeds each photo as it is taken cannot afford a large image tower, and neither can a search service answering thousands of image queries a second. On an iPhone 12 Pro Max, a ViT-L/14 image tower (the Large size, about 304 million parameters) takes 57.9 ms per image. The small MobileCLIP2-S2 tower takes 3.6 ms (Faghri et al. 2025). A small model trained from scratch on web pairs also leaves much accuracy unclaimed, as the numbers below show.

The standard fix is knowledge distillation, training a small student model to copy a large teacher's outputs (§20.4). Here it is applied to the whole batch's similarity matrix instead of one output distribution.

Here is the idea. The teacher, a large trained CLIP or SigLIP model, scores the same batch the student sees. Its row softmax says, for each image, how its probability is spread over the batch's captions. Its column softmax does the same for each caption. A one-hot label says "this caption and no other". The teacher's rows say "mostly this caption, a little of that near-duplicate, none of the rest". Training the student to reproduce those rows and columns passes on what the teacher knows about which captions are close.

📐 Contrastive knowledge distillation

For a batch of $B$ pairs, let $\mathbf S^{\text{te}}, \mathbf S^{\text{st}} \in \R^{B\times B}$ be the teacher's and the student's logit matrices, each with its own logit scale. Let $\tau_{\text{KD}} > 0$ be a distillation temperature. Write $\mathbf A^{\text{te}}$ and $\mathbf A^{\text{st}}$ for the row softmaxes of $\mathbf S^{\text{te}}/\tau_{\text{KD}}$ and $\mathbf S^{\text{st}}/\tau_{\text{KD}}$ , and $\bar{\mathbf A}^{\text{te}}$ and $\bar{\mathbf A}^{\text{st}}$ for their column softmaxes, as in §7.2. The contrastive distillation loss is

\mathcal L_{\text{KD}} = \frac{\tau_{\text{KD}}^2}{2B}\sum_{i=1}^{B}\Big[\operatorname{KL}\big(\mathbf A^{\text{te}}_{i,:}\,\big\Vert\,\mathbf A^{\text{st}}_{i,:}\big) + \operatorname{KL}\big(\bar{\mathbf A}^{\text{te}}_{:,i}\,\big\Vert\,\bar{\mathbf A}^{\text{st}}_{:,i}\big)\Big],

with $\operatorname{KL}(\mathbf p\Vert\mathbf q) = \sum_j p_j\log(p_j/q_j)$ from §0.5. The teacher's distribution comes first, and no gradient flows into the teacher. A student is usually trained on $(1 - \lambda)\,\mathcal L_{\text{CLIP}} + \lambda\,\mathcal L_{\text{KD}}$ for a weight $\lambda \in [0, 1]$ .

In plain words, for every image, make the student's distribution over the batch's captions match the teacher's. For every caption, do the same for its distribution over the images. Each mismatch is measured by the KL divergence from the teacher. The factor $\tau_{\text{KD}}^2$ keeps the gradient from shrinking as the temperature rises, as §20.4 derives. It comes from the original distillation paper (Hinton et al. 2015).

∂ Proof of the distillation gradient

For one row, $\operatorname{KL}(\mathbf p\Vert\mathbf q) = \sum_j p_j\log p_j - \sum_j p_j\log q_j$ , and the first sum does not depend on the student. With $q_j = \operatorname{softmax}(\mathbf s/\tau_{\text{KD}})_j$ , we have $\log q_j = s_j/\tau_{\text{KD}} - \log\sum_k e^{s_k/\tau_{\text{KD}}}$ . Using $\sum_j p_j = 1$ , that gives $\partial(-\sum_j p_j\log q_j)/\partial s_k = -p_k/\tau_{\text{KD}} + q_k/\tau_{\text{KD}}$ . This is the softmax cross-entropy gradient of §0.2 with a soft target.

Each logit $s^{\text{st}}_{ij}$ appears in row $i$ and in column $j$ , so the two terms add. Multiplying by $\tau_{\text{KD}}^2/(2B)$ gives the formula. For a one-hot teacher, $\sum_j p_j\log p_j = 0$ and $-\sum_j p_j\log q_j = -\log A^{\text{st}}_{ii}$ , which is the CLIP row loss, and likewise for the columns. $\square$

In plain words, CLIP is distillation from a teacher that knows only which caption came with which image. A real teacher replaces the identity matrix with its own soft matrices.

That replacement fixes a problem that hard labels cannot. When two captions in a batch are near duplicates, a one-hot label calls one of them a negative. The loss then keeps pushing it away, however good the model is. The teacher puts probability on both, so the student is never pushed away from a caption that also fits. The cell below checks this on six pairs, in which images 1 and 2 are two photos of dogs with the same caption.

Python · cell 10

✏️ Two identical captions, under hard labels and under a teacher

The teacher splits image 1's probability evenly between the two identical captions, as it must. CLIP's hard labels call caption 2 a negative for image 1. So even a student that has learned everything the teacher knows gets a gradient of $-0.052$ on its own caption and $+0.115$ on the identical one. A descent step then lowers the score of a caption that fits perfectly.

No model can do better, since identical captions get identical scores. The rows of images 1 and 2 each cost at least $\log 2$ , a floor of $2\log 2/(2B) = 0.116$ in the loss. The distillation loss of that same student is exactly 0, with a zero gradient, because its targets are the teacher's own split. That loss is minimized only where the student's logits equal the teacher's up to one constant shift.

At real scale, three lines of work set the recipes that interviewers expect you to know. The first matches similarities or features. The second removes the teacher's cost from training. The third tunes the teacher. The box below gives their numbers.

🔬 Distilled image-text encoders, with their numbers

Matching similarities and features. TinyCLIP (Wu et al. 2023) calls the loss above affinity mimicking. It adds weight inheritance, which starts the student from a subset of the teacher's weights. Its ViT-8M/16 student, with an 8-million-parameter image tower, reached 41.1 percent zero-shot ImageNet trained on the 15-million-image YFCC subset, against 37.6 for a ViT-B/16 CLIP trained on the same data. CLIP-KD (Yang et al. 2023) compared several kinds of distillation and found that plain feature mimicry works surprisingly well. That is a mean squared error between the student's embedding, projected to the teacher's width, and the teacher's embedding. With a ViT-L/14 teacher trained on LAION-400M and students trained on the CC3M and CC12M caption sets, a ViT-B/16 student reached 57.5 percent, 20.5 points above the same student without distillation.
Distillation without the teacher's cost. Running a large teacher at every step can cost more than training the student. MobileCLIP (Vasu et al. 2024) moves that cost offline into a reinforced dataset. For every image of DataComp-1B it stores the settings of 10 random augmentations and 5 synthetic captions from a CoCa captioner. It also stores, in 16 bits, the embeddings of each augmented image and of each caption, real and synthetic, from an ensemble of two ViT-L/14 CLIP teachers. Training reads the teachers' embeddings from disk and forms their logits with one matrix product. So an epoch costs the same as plain CLIP training. On its 12-million-image subset and 8 A100 GPUs, an epoch takes 1.3 hours, against 21.1 with the teachers and the captioner run online. The loss is $(1 - \lambda)\mathcal L_{\text{CLIP}} + \lambda\mathcal L_{\text{KD}}$ , averaged over the teachers, with $\lambda = 1$ (distillation alone) for the small S0 to S2 models and 0.75 for the Base one. MobileCLIP-S2 reached 74.4 percent zero-shot ImageNet at 3.6 ms per image and 3.3 ms per text on the phone above. In its ablation (a ViT-B/16 for 30,000 steps on the 12-million-image subset), distillation together with the synthetic captions, the stored augmentations, and the teacher ensemble raised zero-shot ImageNet from 44.5 to 61.7 percent.
Tuning the teacher. MobileCLIP2 (Faghri et al. 2025) swapped in stronger teachers, trained on data selected by the filtering networks of §30.5, and fine-tuned the captioner on smaller sets of high-quality captions. It found that the teacher's logit scale used to form the distillation targets matters. It settled on 60 and 70 for its two teachers, with values within about 5 performing alike. MobileCLIP2-S2 reached 77.2 percent at the same latency. The largest, S4, reached 81.9 percent, matching SigLIP So400m/14 (82.0 in the same table) with about half the parameters.

The MobileCLIP ablation is a reminder, as in §30.5, that what a model is trained on matters as much as how. A distilled student also inherits its teacher's blind spots along with its knowledge, so it is no better on concepts the teacher gets wrong.

Interview

Contrastive distillation loss and its gradient

Implement clip_kd_loss(S_teacher, S_student, tau).

Both inputs are (B, B) logit matrices for the same batch, with rows for images and columns for texts. tau is the distillation temperature.
Return (loss, dS), the loss $\mathcal L_{\text{KD}}$ of the definition and its gradient with respect to S_student. The loss covers rows and columns, puts the teacher first in each KL, and includes the factor $\tau_{\text{KD}}^2$ and the $1/(2B)$ .

Use stable log-softmaxes, since logits can be in the thousands, and no Python loops over entries.

import numpy as np

def clip_kd_loss(S_teacher, S_student, tau):
    """S_teacher, S_student: (B, B) logits, rows images and columns texts. tau: distillation temperature.
    Returns (loss, dS), the contrastive distillation loss over rows and columns and its gradient
    with respect to S_student."""
    B = len(S_student)
    return 0.0, np.zeros((B, B))

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In PyTorch the KL is F.kl_div with log-probabilities for both arguments. Mind its argument order. The student's log-probabilities come first and the teacher's second, and the result is KL(teacher ‖ student).

🔥 In PyTorchfor a machine with PyTorch

The frontier encoders as of late 2026

Two families now compete for the job of embedding images. The first is the pair of towers this chapter has built, trained with a softmax or a sigmoid loss and made better by data, captioning losses, and distillation. Its strongest members in late 2026 are SigLIP 2 and Meta's Perception Encoder, both near 85 percent zero-shot ImageNet. The second family turns a whole vision-language model into an embedder. It feeds an image and a text through one large model and takes a hidden state as the embedding. The box lists the main releases with their dates and numbers.

🔬 Image-text encoders at the frontier, with dates

Open recipes before SigLIP 2. EVA-CLIP (Sun et al. 2023, March) started its image tower from a model pretrained by masked image modeling, and its text tower from an earlier CLIP. It trained with LAMB, an optimizer built for very large batches, at batches up to 144,000. It also dropped 50 percent of the image patches at random, a trick that §30.2 covers. Its EVA-02-CLIP-E/14+ (5 billion parameters) reached 82.0 percent zero-shot ImageNet after 9 billion samples. EVA-CLIP-18B (Sun et al. 2024, February) scaled the same recipe to a 17.5-billion-parameter image tower, itself first pretrained to reconstruct the features of the smaller EVA-CLIP. On a mix of 2 billion public pairs drawn from LAION-2B and COYO-700M, it reached 83.8 percent on ImageNet, and an average of 80.7 over 27 classification benchmarks, after about 6 billion samples.
SigLIP 2 (February 2025), above.
Perception Encoder (Bolya et al. 2025, April, from Meta). PE-Core G has a 1.9-billion-parameter vision tower that reaches 85.4 percent zero-shot ImageNet at 448 px. It was trained with CLIP's softmax loss at a global batch of 131,072, with LAMB, and with a resolution that grew from 98 to 448 px. Its data were 5.4 billion pairs, curated by balancing over a list of concepts as MetaCLIP does (§30.5), and it saw 86 billion samples in all. Its title states its most interesting finding, that the best features for dense tasks and for VLMs sit in intermediate layers rather than at the output.
Meta CLIP 2 (Chuang et al. 2025, July) extends that curation to worldwide data in over 300 languages (§30.5).
Small encoders. MobileCLIP (November 2023) and MobileCLIP2 (August 2025), distilled as in the previous part. MobileCLIP2's image towers run from 1.5 ms per image on an iPhone 12 Pro Max at 71.5 percent zero-shot ImageNet (S0) to 19.6 ms at 81.9 percent (S4).
Encoders for VLMs that drop the contrastive text tower. The captioning-trained AIMv2 (November 2024) and OpenVision 2 (September 2025) compete as vision towers for VLMs. So do encoders trained without any language, Web-SSL (Fan et al. 2025, April) and DINOv3 (Siméoni et al. 2025, August).
Embedders built on vision-language models. VLM2Vec (Jiang et al. 2024, October) turns a VLM into an embedder. It writes an instruction in front of the input and takes the final hidden state of the last token. It also introduced the MMEB benchmark, 36 datasets across classification, visual question answering, retrieval, and grounding. There its best model scored 62.9, against 47.2 for the best fine-tuned CLIP-style baseline. LLM2CLIP (Huang et al. 2024, November) keeps a pretrained CLIP image tower and trains it further. It replaces the text tower with Llama 3.1 8B, first fine-tuned contrastively on captions and then frozen behind a small adapter. GME (Zhang et al. 2024, December, on Qwen2-VL), jina-embeddings-v4 (Günther et al. 2025, June, 3.8 billion parameters, with single-vector and multi-vector outputs), and Qwen3-VL-Embedding (Li et al. 2026, January, 2 and 8 billion parameters, 77.8 overall on MMEB-V2 at release) followed. Google's closed Gemini Embedding 2 (Shanbhogue et al. 2026, May) embeds video and audio in the same space.

Two cautions go with any such list. First, "best" depends on the use. PE-Core G edges SigLIP 2 g on zero-shot ImageNet (85.4 against 85.0 at 384 px) in PE's own table. But captioning-trained and self-supervised encoders often win as VLM backbones and on dense tasks, where zero-shot classification is not the measure. And VLM-based embedders win on MMEB-style tasks that a pair of contrastive towers was never trained for. Second, the list ages quickly. The 2026 releases we checked through October were mostly VLM-based embedders and distilled small encoders rather than new contrastive towers. Check the latest releases and their evaluation protocols before you say which is best.

The VLM-based embedders are where the field moved in 2025 and 2026. Such an embedder is the decoder-LLM embedder of §7.3 with images among its inputs, and it inherits that section's habits. It pools the last token and takes an instruction written in front of each query. It often trains its embeddings so that a prefix of the coordinates works on its own, a property called Matryoshka embeddings. It also pays that section's price. Every embedding costs a forward pass through a model of billions of parameters, against about 0.4 billion for a SigLIP 2 So400m image tower.

In return it reads text inside images and follows instructions. It accepts an image and a text together as one query ("this jacket, in green"). And it handles long, compositional queries that a 64-token text tower truncates. It has become the usual choice for retrieval over documents, screenshots, and mixed media. Pairs of towers remain the usual choice for high-volume photo search, zero-shot classification, and the vision side of VLMs.

The Perception Encoder's recipe ablation also bears on this section's main question. On a fixed set of 2.3 billion pairs, it used compute saved elsewhere to double the softmax batch from 32,768 to 65,536, which also doubled the samples seen. That added 0.6 points of ImageNet accuracy and 1.1 points of average robustness. The gain mixes a larger batch with more data, but it shows the softmax still improving past 32,768. With enough memory and a large enough batch, the softmax is not a handicap. That leads to the practical choice.

Softmax or sigmoid

	Softmax (CLIP, CoCa, PE)	Sigmoid (SigLIP, SigLIP 2)
What each logit answers	which of the $B$ candidates is mine	does this one pair match
Normalizer	log-sum-exp over every row and every column	none, one learned bias
Logits per device, sharded	$2bB$ (row and column slices)	$b^2$ (one block at a time)
Communication per step, forward	two all-gathers, $2(D-1)bd$ numbers	$D-1$ neighbor permutes, $(D-1)bd$
Quality below a batch of 16k	worse	better
Quality at 32k and above	about equal, keeps improving to 64k in PE's setting	about equal, saturates near 32k
Extra state and tuning	learned scale	learned scale and bias, bias must start near $-\log(B-1)$
Score meaning	PMI up to an unknown constant, not a probability	PMI minus $\log(B-1)$ , a match probability under the training prior $1/B$ , re-biased for other base rates
Label noise	less robust in SigLIP's tests	more robust

Choose the sigmoid when memory per device limits the batch, or when there are many devices and a global gather is expensive. Choose it too when you want one threshold on match scores, or when training is small or noisy. Choose the softmax when you have the memory for very large batches and an existing softmax pipeline, as the Perception Encoder did. At large batches the evidence says that the choice of data (§30.5) matters far more than the choice of loss.

⚠️ Common ways to get SigLIP wrong

Starting the bias at 0. The first steps then push every pair apart, which can wreck a large-batch run before it learns anything.
Computing the scale gradient as the sum of $G_{ij}s_{ij}$ . The bias is not scaled, so the derivative of each logit with respect to $\alpha$ is $s_{ij} - b_0$ .
Normalizing by $B^2$ instead of $B$ . That divides every gradient by the batch size, which lowers the learning rate under plain gradient descent and, under Adam, changes the meaning of gradient clipping and the weight against any other loss.
Putting positive labels on the diagonal of every block of the ring. Only the block of a device's own texts holds positives.
Fine-tuning an unlocked pretrained tower with the default weight decay, which erodes the representation you meant to keep.
Leaving duplicate captions in a batch ("a photo of a dog"). They become false negatives with confident pushes, a problem the softmax shares.

🧭 What to remember

SigLIP treats each of the $B^2$ image-text pairs as its own logistic regression, $\mathcal L = -\frac1B\sum_{i,j}\log\sigma(z_{ij}(e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j + b_0))$ . Each negative's push $\sigma(s_{ij})$ depends on that pair alone, while a softmax weight is a sigmoid whose threshold is the log-sum-exp of the whole row.
With all cosines near 0, the loss per image is minimized at $b_0 = -\log(B-1)$ , about $-10.4$ at a batch of 32k. There the push of the negatives balances the pull of the positive. Starting at 0 over-corrects, and SigLIP starts at $b_0 = -10$ and $e^{\alpha} = 10$ .
The population optimum is $\operatorname{PMI} - \log(B-1)$ at every batch size, so the bias absorbs the batch's prior and the scaled cosine targets PMI. That is the precise sense of decoupling. The prior itself still moves with $B$ , so $\sigma(s)$ is a match probability under the prior $1/B$ . Adding $\log(B-1) + \log\frac{\pi}{1-\pi}$ to the logit re-sets it for a deployment base rate $\pi$ .
The sigmoid beats the softmax below a batch of about 16k and saturates near 32k. The softmax catches up only at larger batches. At large batches training needs $\beta_2 = 0.95$ to stay stable.
The chunked ring scores one $b\times b$ block at a time with neighbor permutes. It holds $b^2$ logits instead of $2bB$ and sends half the bytes of two all-gathers.
Freezing a strong pretrained image tower and training only the text tower (locked-image tuning) is cheap and often better than unfreezing it. Captioning losses give encoders richer patch features, and SigLIP 2 combines both ideas with the sigmoid loss.
Small, fast encoders come from distillation, which matches the student's row and column softmaxes to a teacher's by $\operatorname{KL}(\text{teacher}\Vert\text{student})$ . Its soft targets also stop near-duplicate captions from being pushed apart. MobileCLIP stores the teachers' outputs offline so that distillation costs no more than plain training, and VLM-based embedders now lead on retrieval over documents and with instructions. Next, §30.5 shows why, at a fixed compute budget, the data decides more than any of these choices.

Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training. The loss, its bias, the chunked implementation in Section 3.3, the batch-size study up to one million, and the ablations on bias, negatives, noise, and $\beta_2$ .
Zhai et al. (2022). LiT: Zero-Shot Transfer with Locked-image text Tuning. The L, U, u study and the measurements behind why locking generalizes better.
Tschannen et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. The combined recipe, NaFlex, and Table 1's comparison with SigLIP at every size.
Yu et al. (2022). CoCa: Contrastive Captioners are Image-Text Foundation Models. One forward pass for a contrastive and a captioning loss, with the split decoder.
Tschannen et al. (2023). Image Captioners Are Scalable Vision Learners Too. The matched comparison of captioning and contrastive pretraining, and parallel prediction.
Bolya et al. (2025). Perception Encoder: The best visual embeddings are not at the output of the network. A softmax-trained encoder at a batch of 131k, its recipe ablations, and where its best features live.
Vasu et al. (2024). MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training. Contrastive distillation from a teacher ensemble with the teachers' outputs and synthetic captions stored offline, and the latency-accuracy trade-off measured on a phone. Its sequel, MobileCLIP2: Improving Multi-Modal Reinforced Training (Faghri et al., 2025), updates the teachers and tunes their temperature.
Jiang et al. (2024). VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. How a VLM becomes an instruction-following embedder, and the MMEB benchmark that compares it with CLIP-style towers.

Made for larger screens

SigLIP and better losses

One yes-or-no question per pair

A two-by-two example by hand

The gradient, pair by pair

SigLIP loss and gradients

Softmax and sigmoid weights on the negatives

Why the bias starts near minus log B

The right initial bias, with and without spread

What a perfectly trained score measures

What the sigmoid of a trained score means

What decoupling means, and what it does not

What the batch size does, measured

Computing the loss on a ring of devices

The chunked ring

In PyTorch

Locking the image tower

Locked-image tuning on frozen features

Adding a captioning loss

SigLIP 2 combines the pieces

Distilling a joint encoder

Contrastive distillation loss and its gradient

The frontier encoders as of late 2026

Softmax or sigmoid

Read more

Watch next