Chapter 28 · Section 28.4 · 95 min

Self-distillation with DINO

This section is about self-distillation, a way to train an image encoder with no labels by teaching a network to agree with a slowly averaged copy of itself. The network sees two different crops of the same photograph. The copy describes one crop, and the network is trained to give the same description of the other crop. The copy's weights are a running average of the network's own past weights, so it changes slowly. Nothing in the loop ever says what is in the photograph.

The method that made this work is DINO (Caron et al. 2021), and its results surprised the field. Its features classify ImageNet by nearest neighbors almost as well as a trained linear classifier does. Its attention maps outline objects that nobody ever marked. Its successors, DINOv2 and DINOv3, are among the strongest image encoders trained without text or labels. They are what you reach for when you need frozen features for segmentation, depth, retrieval, or a robot's camera, and vision teams expect you to explain why they work.

The idea starts from knowledge distillation (§20.4). There, a large network that already knows the task, the teacher, turns each input into a probability distribution over classes, softened by dividing its logits by a temperature. These soft targets say how the teacher ranks the wrong answers as well as the right one. A smaller network, the student, is trained to reproduce them. Without labels no trained teacher exists. Self-distillation builds one out of the student itself, and the student learns by predicting what the teacher says about a different crop.

Two things about this should puzzle you. The teacher is made of nothing but past students, yet it is more accurate than the student at every point of training. And the game has a cheat. A student and a teacher that give every image the same output agree perfectly and have learned nothing. This failure is called collapse, and §28.3 showed that it is the central danger of any method that only pulls two views of an image together. The methods there escape it in two ways. Some make the two branches different, training the network to predict a slowly moving copy of itself through an extra small predictor. Others add a term to the loss that keeps the outputs spread apart, for example by forcing each batch to use many clusters. DINO keeps the slowly moving copy and replaces everything else with two one-line operations on its output, centering and sharpening.

This section answers four questions. Why is an average of past students a better teacher than the latest student? Why does DINO need both centering and sharpening, and what goes wrong with only one? What do small crops add, and what do they cost? And how did DINOv2 and DINOv3 turn the recipe into the frontier of self-supervised vision?

Two conventions run through the section. Most experiments use a Vision Transformer (ViT), which cuts an image into square patches, treats each patch as a token, and adds one learned class token whose output summarizes the image (§27.1). ViT-S/16 is the small model, with about 22 million parameters and patches 16 pixels on a side, and ViT-B is the base model with about 86 million. Two numbers grade a frozen encoder, both from §28.1. Linear-probe accuracy trains one linear classifier on the frozen features of the labeled training images and reports its top-1 accuracy on the validation set. k-NN accuracy trains nothing and labels each validation image by a vote of its nearest training images in feature space. Unless a sentence says otherwise, both are measured on ImageNet-1k, 1.28 million photographs in 1,000 classes.

A teacher built from the student

Start with the distillation loss in symbols. A network ends in a vector of logits $\mathbf{z}$ , one raw score per output. A softmax at temperature $\tau$ turns them into probabilities, $\mathbf{p} = \operatorname{softmax}(\mathbf{z}/\tau)$ . A temperature below 1 makes the distribution sharper, and one above 1 makes it flatter. For logits $(2, 1, 0)$ , temperature 1 gives probabilities $(0.665, 0.245, 0.090)$ , and temperature 0.5 gives $(0.867, 0.117, 0.016)$ .

The student is trained to minimize the cross-entropy from the teacher's distribution $\mathbf{p}_t$ to its own distribution $\mathbf{p}_s$ ,

H(\mathbf{p}_t, \mathbf{p}_s) = -\sum_k p_{t,k}\log p_{s,k} .

In plain words, the loss is small when the student puts high probability where the teacher does. The teacher's own entropy does not depend on the student, so minimizing this is the same as minimizing the KL divergence from teacher to student (§0.5).

Self-distillation keeps that loss and changes where the teacher comes from. The student has parameters $\theta$ and is trained by gradient descent. The teacher has the same architecture and its own parameters $\xi$ , which never receive a gradient. Instead, after every optimizer step, the teacher's weights move a small step toward the student's. With $m = 0.996$ , the new teacher keeps 99.6 percent of the old teacher and mixes in 0.4 percent of the current student,

\xi \leftarrow m\,\xi + (1 - m)\,\theta, \qquad 0 < m < 1 .

This update is an exponential moving average (EMA) of the student's weights, and $m$ is called the momentum. It is the same average that drives the slowly updated second encoder of the contrastive method MoCo in §28.2, and the weight averaging that §0.6 uses for evaluation. Unrolled, it gives the student from $s$ steps ago the weight $(1 - m)m^s$ . These weights fall off geometrically. So the teacher averages over a window of about $1/(1 - m)$ steps, and the mean age of the students in it is $m/(1 - m)$ steps, as §28.2 derives.

✏️ How far back the teacher remembers

DINO starts at $m = 0.996$ . Its teacher is an average over about $1/0.004 = 250$ steps, with a mean lag of $0.996/0.004 = 249$ steps and a half-life of $\ln 0.5/\ln 0.996 \approx 173$ steps. With a batch of 1,024 ImageNet images, one epoch is $1{,}281{,}167/1{,}024 \approx 1{,}251$ steps, so the teacher summarizes roughly the last fifth of an epoch.

DINOv2 starts at $m = 0.994$ , a window of about 167 steps. DINOv3 keeps $m = 0.999$ fixed for its whole run, a window of 1,000 steps. DINO and DINOv2 raise $m$ toward 1 along a cosine, $m(t) = 1 - (1 - m_0)\big(1 + \cos(\pi t/T)\big)/2$ at step $t$ of $T$ , starting from $m_0$ . The teacher therefore freezes as training ends.

Why should an average of students be better than the latest one? An optimizer with a fixed step size never settles at a solution. Each minibatch gives a slightly different gradient, so the weights jitter around a good point. Averaging the recent weights keeps the center of the jitter and cancels much of the noise. §0.6 proves this on a noisy quadratic, where an average whose window is ten times the optimizer's own memory removes about 90 percent of the excess loss. Averaging an optimizer's iterates is called Polyak-Ruppert averaging. Caron et al. read their teacher exactly this way, as an ensemble of recent students that beats any one of them.

The loop then feeds itself. The student chases a target better than itself, so the student improves, so the next average improves. A weight-averaged teacher had already been used in semi-supervised learning, where some labels exist, as the mean teacher of Tarvainen and Valpola (2017). DINO uses it with no labels at all.

The previous-epoch row deserves a second look. A teacher that changes only once per epoch is slow enough not to collapse. It already matches two strong methods from earlier sections, which the DINO authors retrained on the same ViT for 300 epochs (the paper's Table 7). MoCo v2, a contrastive method that compares each image against a queue of other images, reaches 62.0 percent k-NN. BYOL, which predicts a moving-average target through an extra small network, reaches 66.6. A smooth average of the recent past does better still, at 72.8. A teacher equal to the current student is the fast extreme, and the next sections explain exactly why it fails.

In practice, the teacher's momentum should err on the slow side. A teacher that follows the student too closely collapses the run, while one that changes once per epoch still works, only less well. An interviewer will ask why a teacher made of past students can beat the student, and the answer is the jitter-averaging argument above.

DINO, step by step

Here is one training step in plain words, before any symbols.

Cut each photograph into crops. Two are large global crops, and several are small local crops. Each crop is randomly recolored, blurred, and sometimes solarized (its bright pixels inverted), as in §28.2.
The teacher looks at the two global crops only. The student looks at every crop.
Each network ends in a head, a small network that outputs $K_p = 65{,}536$ scores. Each score belongs to a prototype, a learned vector that acts like an unnamed category, and it measures how well the crop matches that prototype. The clustering method SwAV in §28.3 uses the same device.
The teacher subtracts a running average of its own scores (centering), divides by a small temperature (sharpening), and applies a softmax.
The student divides its scores by a larger temperature and applies a softmax.
The loss is the cross-entropy between the teacher's distribution for one crop and the student's distribution for a different crop of the same image, averaged over all such pairs.
The student takes a gradient step. Then the teacher's weights and the running average each take an EMA step.

The box states the method exactly. The rest of the section takes it apart, and DINOv2 and DINOv3 later replace some of its pieces.

📐 DINO

Let $x$ be an unlabeled image from the training set $\mathcal{D}$ . From it draw $V_g = 2$ global crops $x^g_1, x^g_2$ and $V_l$ local crops (6 to 10 in the paper), $V = V_g + V_l$ crops in all, each with the color, blur, and solarization augmentations of §28.2.

The student is a backbone $f_\theta$ followed by a head $g_\theta$ . The backbone is a ViT whose class-token output $\mathbf{h} = f_\theta(x) \in \R^d$ is the representation used downstream. The head is a 3-layer MLP with GELU activations and hidden width 2048. Its 256-dimensional output is $\ell_2$ -normalized and then mapped to $K_p = 65{,}536$ outputs by a linear layer without bias whose rows are rescaled to unit length (weight normalization). For a crop $x'$ the student logits are $\mathbf{z}_s = g_\theta(f_\theta(x')) \in \R^{K_p}$ . The teacher has the identical architecture with parameters $\xi$ , and for a global crop $x^g$ its logits are $\mathbf{z}_t = g_\xi(f_\xi(x^g))$ . The two become probability vectors,

\mathbf{p}_s = \operatorname{softmax}\!\Big(\frac{\mathbf{z}_s}{\tau_s}\Big), \qquad \mathbf{p}_t = \operatorname{softmax}\!\Big(\frac{\mathbf{z}_t - \mathbf{c}}{\tau_t}\Big),

with student temperature $\tau_s = 0.1$ , teacher temperature $\tau_t$ raised linearly from 0.04 to 0.07 over the first 30 epochs, and a center $\mathbf{c} \in \R^{K_p}$ . For a batch of $B$ images, the loss averages the cross-entropy $H(\mathbf{p}, \mathbf{q}) = -\sum_k p_k\log q_k$ over every pair of a global crop seen by the teacher and a different crop seen by the student,

\mathcal{L}(\theta) = \frac{1}{V_g(V - 1)}\sum_{i=1}^{V_g}\ \sum_{\substack{v=1\\ v\ne i}}^{V}\ \frac{1}{B}\sum_{b=1}^{B} H\big(\mathbf{p}_t(x^g_{b,i}),\ \mathbf{p}_s(x_{b,v})\big).

After each gradient step on $\theta$ (no gradient flows to $\xi$ or $\mathbf{c}$ ), the teacher and the center are updated,

\xi \leftarrow m\,\xi + (1 - m)\,\theta, \qquad \mathbf{c} \leftarrow m_c\,\mathbf{c} + (1 - m_c)\,\frac{1}{V_gB}\sum_{b=1}^{B}\sum_{i=1}^{V_g}\mathbf{z}_t(x^g_{b,i}),

with $m$ on a cosine schedule from 0.996 to 1 and center momentum $m_c = 0.9$ . The optimizer is AdamW with batch 1,024, learning rate $0.0005\times B/256$ after a 10-epoch linear warmup, then cosine decay, and weight decay on a cosine from 0.04 to 0.4. Both networks start from the same initialization.

In plain words, the teacher looks at a big crop of the photo and writes down a distribution over 65,536 made-up categories. Before the softmax it subtracts a running average of what it usually writes, and it exaggerates the differences that remain. The student looks at another crop, often a tiny one, and is graded on how well it reproduces that distribution. The student learns, the teacher drifts slowly after it, and nothing ever names a real category.

Here $\mathbf{h}$ is the representation you keep and $\mathbf{z}$ is the head output the loss sees, the convention of §28.2. The head is thrown away after training, like the projection head of every method in this chapter.

The student's gradient has the same form as any softmax classifier's (§0.2), with the teacher's distribution in place of a one-hot label.

Proof. Write $\log p_{s,k} = z_{s,k}/\tau_s - \log\sum_j e^{z_{s,j}/\tau_s}$ , so $\partial\log p_{s,k}/\partial z_{s,j} = (\mathbb{1}[k = j] - p_{s,j})/\tau_s$ . Then $\partial H/\partial z_{s,j} = -\sum_k p_{t,k}(\mathbb{1}[k = j] - p_{s,j})/\tau_s = (p_{s,j} - p_{t,j})/\tau_s$ , using $\sum_k p_{t,k} = 1$ . The teacher's parameters enter only through $\mathbf{p}_t$ , which is treated as a constant. ∎

In plain words, a gradient step lowers each student score where the student gives that prototype more probability than the teacher does, and raises it where the student gives less. The push stops exactly when the two distributions are equal. Because $\tau_t < \tau_s$ , the teacher's distribution is sharper than the one the student would produce from the same centered logits. So even a student identical to its teacher keeps being pushed toward a more confident version of itself. That push is what the next section is about.

Two ways to collapse, and the two cures

Collapse, for DINO, means the teacher gives every image the same target. Suppose the teacher assigns every image the target $\mathbf{q}$ and the student outputs $\mathbf{q}$ for every crop. Then $\mathbf{p}_s - \mathbf{p}_t = \mathbf{0}$ for every pair. The student's gradient is zero, the student stops moving, and the teacher converges to it. Nothing in the loss rules this out. A constant output is a fixed point of the whole system, and the representation behind it is useless. This is complete collapse in the language of §28.3.

DINO meets it in two flavors. In the first, one prototype wins for every image, so every target is the same one-hot vector. In the second, every target is the flat, uniform distribution. Splitting the loss into two pieces shows that the student cannot tell either from success.

∴ Cross-entropy is teacher entropy plus KL

For any probability vectors $\mathbf{p}_t, \mathbf{p}_s$ over $K_p$ outcomes,

H(\mathbf{p}_t, \mathbf{p}_s) = H(\mathbf{p}_t) + D_{\mathrm{KL}}(\mathbf{p}_t\,\Vert\,\mathbf{p}_s), \qquad 0 \le H(\mathbf{p}_t) \le \log K_p .

The student controls only the KL term. If the teacher's target does not depend on the image, a student that outputs it makes the KL zero for every pair. The two extreme constant targets are a one-hot vector, with $H(\mathbf{p}_t) = 0$ (one dimension wins for every image), and the uniform vector, with $H(\mathbf{p}_t) = \log K_p$ (no dimension wins for any image). Both give a KL of zero and a useless representation.

Proof. $H(\mathbf{p}_t, \mathbf{p}_s) = -\sum_k p_{t,k}\log p_{s,k} = -\sum_k p_{t,k}\log p_{t,k} + \sum_k p_{t,k}\log(p_{t,k}/p_{s,k})$ , which is the identity of §0.5. Entropy is largest for the uniform vector and smallest, zero, for any one-hot vector. ∎

So the loss cannot detect collapse. We need a number that can. The right one asks how much a target tells you about which image produced it.

In plain words, a healthy run wants $H(\bar{\mathbf{p}}_t)$ large, meaning all prototypes get used across the dataset, and $\mathbb{E}_x H(\mathbf{p}_t(x))$ small, meaning each image commits to a few of them. Both flavors of collapse give $I = 0$ . Centering raises the first term, and sharpening lowers the second. A small example shows each operation at work.

✏️ Three images and four prototypes, by hand

Take $K_p = 4$ and three images whose teacher logits are $(2, 1, 0, 0)$ , $(2, 0, 1, 0)$ , and $(2, 0, 0, 1)$ . They share a large first coordinate, the kind of input-independent component a network produces early in training, and differ only in where they put a 1.

No centering. At $\tau_t = 0.1$ , the first coordinate leads by $1/0.1 = 10$ after dividing by the temperature. Every target puts probability $1 - 4.5\times10^{-5}$ on prototype 1, with entropy 0.0005. All three images get the same one-hot target. One dimension has won.

Centering. The batch mean is $\mathbf{c} = (2, \tfrac13, \tfrac13, \tfrac13)$ . After subtracting it, the first image's logits are $(0, \tfrac23, -\tfrac13, -\tfrac13)$ , and the others are its permutations. At $\tau_t = 0.1$ the targets put probability 0.9986 on prototypes 2, 3, and 4 respectively, entropy 0.011 each. Each image now has its own target.

Centering without sharpening. At $\tau_t = 1$ the same centered logits give the soft target $(0.228, 0.445, 0.164, 0.164)$ , entropy 1.290, close to the uniform $\log 4 = 1.386$ . Centering kept the images apart, and only the low temperature kept their targets from melting into the uniform vector.

The cell reproduces these numbers and prints the information $I$ in each case.

Python · cell 1

Only the centered, sharpened targets carry real information, $I = 1.096$ nats, almost the $\log 3 = 1.099$ that three fully distinct targets would carry. The other three cases carry 0.1 nats or less.

The two operations do opposite things. Centering subtracts whatever the teacher says about every image. That removes the dominant dimension, but it leaves only the small image-specific differences, so on its own it pushes targets toward uniform. Sharpening divides those differences by a small temperature. That makes each target confident, but on uncentered logits it also makes the shared dimension win everywhere. Each operation cures the other's failure.

Caron et al. measured exactly this in their collapse study (Figure 7). With either operation removed, the KL between teacher and student goes to zero. The teacher's entropy goes to 0 without centering and to $\log K_p$ without sharpening.

What centering removes, exactly

The first cure can be stated exactly. Think of the teacher's logits as a part shared by every image plus a part specific to each one.

∴ Centering removes input-independent logits

Suppose the teacher's logits on a batch are $\mathbf{z}_t(x_b) = \mathbf{a} + \mathbf{b}(x_b)$ for $b = 1, \dots, B$ , a shared vector $\mathbf{a} \in \R^{K_p}$ plus an image-specific part.

With the exact batch center $\mathbf{c} = \frac1B\sum_b\mathbf{z}_t(x_b)$ , the centered logits are $\mathbf{b}(x_b) - \bar{\mathbf{b}}$ with $\bar{\mathbf{b}} = \frac1B\sum_b\mathbf{b}(x_b)$ . So the targets $\mathbf{p}_t$ do not depend on $\mathbf{a}$ at all.
With the running center $\mathbf{c}_n = m_c\mathbf{c}_{n-1} + (1 - m_c)\bar{\mathbf{z}}_n$ , suppose the shared vector has equaled $\mathbf{a}$ at steps $1, \dots, n$ and the center was $\mathbf{c}_0$ before step 1. Then $\mathbf{z}_t(x) - \mathbf{c}_n = m_c^n(\mathbf{a} - \mathbf{c}_0) + \mathbf{r}_n(x)$ , where $\mathbf{r}_n(x) = \mathbf{b}_n(x) - (1 - m_c)\sum_{j=1}^{n}m_c^{n-j}\bar{\mathbf{b}}_j$ does not involve $\mathbf{a}$ . The shared vector survives centering only with weight $m_c^n$ .

The official code computes each step's targets with the center from before that step's update, $\mathbf{c}_{n-1}$ . The same statement then holds with $m_c^{n-1}$ in place of $m_c^n$ .

∂ Proof of the centering theorem

Part 1 is one line, $\mathbf{z}_t(x_b) - \mathbf{c} = \mathbf{a} + \mathbf{b}(x_b) - \mathbf{a} - \bar{\mathbf{b}}$ .

For part 2, unroll the recursion with $\bar{\mathbf{z}}_j = \mathbf{a} + \bar{\mathbf{b}}_j$ , giving $\mathbf{c}_n = m_c^n\mathbf{c}_0 + (1 - m_c)\sum_{j=1}^n m_c^{n-j}(\mathbf{a} + \bar{\mathbf{b}}_j)$ . The geometric weights on $\mathbf{a}$ sum to $(1 - m_c)\sum_{j=1}^n m_c^{n-j} = 1 - m_c^n$ . So $\mathbf{z}_t(x) - \mathbf{c}_n = \mathbf{a} - (1 - m_c^n)\mathbf{a} - m_c^n\mathbf{c}_0 + \mathbf{r}_n(x)$ , which is the claim. ∎

In plain words, the center is a running estimate of what the teacher says about every image, and subtracting it deletes that from the target. A shared component that appeared $n$ steps ago is still present with weight $m_c^n$ . With $m_c = 0.9$ the center averages over about 10 steps, and a new shared component falls below 1 percent within $\ln 0.01/\ln 0.9 \approx 44$ steps. With $m_c = 0.999$ it takes about 4,600 steps, plausibly long enough for one dimension to win first. The paper's run with $m_c = 0.999$ did collapse (k-NN 0.1 percent), while $m_c$ of 0, 0.9, and 0.99 gave 69.1, 69.7, and 69.4 percent in short ViT-S ablations (Appendix D).

Centering uses only a mean over the batch. That makes it a cheap relative of the balancing step of SwAV in §28.3, which rescales each batch's scores with the Sinkhorn-Knopp algorithm so that every prototype receives the same total share. DINO's mean only stops any one prototype from being favored by every image. An interviewer may ask why a mean is enough. The answer is part 1 of the theorem, since the cheat DINO must block is exactly a shift shared by all images.

Why sharpening is needed on top

Centering alone pushes the targets toward uniform, and a simple model says how fast. Assume the logits are small, so the softmax is close to linear in them. Then one round of distillation, in which the student learns its best response and the teacher catches up, scales the image-specific part of the logits by a single number. Whether that number is above or below 1 decides between health and collapse.

∴ A first-order model of the collapse dynamics

Assume the logits are small enough that softmax is linear in them. Write the teacher's logits as $\mathbf{z}_t(x) = \mathbf{a} + \delta\,\mathbf{b}(x)$ , a shared part plus an image-specific part of size $\delta$ , with $\mathbb{E}_x\mathbf{b}(x) = \mathbf{0}$ . Let the student be flexible enough to output the best response to its targets, and let the teacher catch up with the student between rounds. Suppose the image-specific part of one crop predicts that of another linearly, $\mathbb{E}[\mathbf{b}(x) \mid x'] = \alpha\,\mathbf{b}(x')$ for a crop $x'$ of the same image, with $0 \le \alpha \le 1$ . Then one round of distillation maps

\delta \;\mapsto\; \alpha\,\frac{\tau_s}{\tau_t}\,\delta, \qquad \mathbf{a} \;\mapsto\; \begin{cases} \mathbf{0} & \text{with exact centering,}\\[2pt] \dfrac{\tau_s}{\tau_t}\,\mathbf{a} & \text{without centering,}\end{cases}

up to vectors with equal entries, which softmax ignores.

∂ Derivation of the first-order model

Best response. For a fixed teacher, the student's expected loss on crop $x'$ is $\mathbb{E}[H(\mathbf{p}_t(x), \mathbf{p}_s(x')) \mid x']$ . Cross-entropy is linear in its first argument, so this equals $H(\mathbb{E}[\mathbf{p}_t \mid x'], \mathbf{p}_s)$ . By the split into entropy plus KL, it is minimized over $\mathbf{p}_s(x')$ by the conditional mean of the target, $\mathbf{p}_s(x') = \mathbb{E}[\mathbf{p}_t(x) \mid x']$ .

Linearized softmax. §20.4 showed that near zero, $\operatorname{softmax}(\mathbf{y}) = \frac{1}{K_p}\mathbf{1} + \frac{1}{K_p}\mathbf{C}\mathbf{y} + O(\lVert\mathbf{y}\rVert^2)$ , where $\mathbf{C} = \mathbf{I} - \frac{1}{K_p}\mathbf{1}\mathbf{1}^\top$ removes the mean of a vector's entries.

With centering. The exact center is $\mathbf{a}$ , since the batch mean of $\mathbf{b}$ is zero in the model. So $\mathbf{p}_t(x) \approx \frac{1}{K_p}\mathbf{1} + \frac{\delta}{K_p\tau_t}\mathbf{C}\mathbf{b}(x)$ , and its conditional mean given $x'$ is $\frac{1}{K_p}\mathbf{1} + \frac{\alpha\delta}{K_p\tau_t}\mathbf{C}\mathbf{b}(x')$ . A student logit vector $\mathbf{z}_s(x')$ gives $\mathbf{p}_s(x') \approx \frac{1}{K_p}\mathbf{1} + \frac{1}{K_p\tau_s}\mathbf{C}\mathbf{z}_s(x')$ . Matching the two, $\mathbf{C}\mathbf{z}_s(x') = \alpha\,(\tau_s/\tau_t)\,\delta\,\mathbf{C}\mathbf{b}(x')$ . The student's new logits have no shared part and an image-specific part scaled by $\alpha\tau_s/\tau_t$ . When the teacher catches up, those are the teacher's logits for the next round.

Without centering. Now $\mathbf{p}_t(x) \approx \frac{1}{K_p}\mathbf{1} + \frac{1}{K_p\tau_t}\mathbf{C}(\mathbf{a} + \delta\mathbf{b}(x))$ , and the same matching gives $\mathbf{C}\mathbf{z}_s(x') = (\tau_s/\tau_t)\,\mathbf{C}\big(\mathbf{a} + \alpha\,\delta\,\mathbf{b}(x')\big)$ . The shared part is multiplied by $\tau_s/\tau_t$ and the image-specific part by $\alpha\tau_s/\tau_t$ .

The model stops being valid once the logits grow large enough for the softmax to saturate, which is where real training settles. It describes which way the system moves from small logits, as at initialization. ∎

Here $\alpha$ measures how much one crop can predict about another. Augmentation is what makes it less than 1, because a crop cannot predict everything about a different crop. Read the three regimes off the map.

Centering and sharpening, with $\alpha\,\tau_s/\tau_t > 1$ . The image-specific part grows each round until the softmax saturates into confident, different targets for different images. This is the healthy outcome.
Centering without sharpening, $\tau_t \ge \tau_s$ . The factor is $\alpha\tau_s/\tau_t \le \alpha < 1$ , so the image-specific part shrinks geometrically and the targets go uniform. Whatever one crop cannot predict from another is averaged away, round after round.
No centering. The shared part grows by $\tau_s/\tau_t$ and the image-specific part only by $\alpha\tau_s/\tau_t$ . The ratio of shared to specific grows by $1/\alpha$ each round, and one dimension wins.

The model also says that sharpening has to beat augmentation, $\tau_t < \alpha\,\tau_s$ . The paper's temperature sweep has that shape.

In the model, success at 0.06 and failure at 0.08 would mean $\alpha$ between 0.6 and 0.8. That is a consistency check on a simplified model, not a measurement. The other two numbers teach something too. An argmax teacher ( $\tau_t \to 0$ ) gives hard one-hot targets that throw away the teacher's uncertainty, and accuracy drops 26 points. And a teacher temperature above 0.06 does not collapse if training starts lower and raises it, which is why the recipe warms $\tau_t$ up. In practice, if a DINO run's loss sits at $\ln K_p$ , the first thing to check is the teacher temperature.

The widget below runs the same story on a batch of 32 images. The teacher's logits are a shared vector plus a small image-specific part. You choose the centering and the teacher temperature, and Play runs the center forward step by step as it absorbs the shared part.

Warm-up

Centering and sharpening

Write the teacher's side of DINO, the two operations that keep it from collapsing.

teacher_probs(z_t, c, tau_t) takes teacher logits z_t of shape (n, K_p), one row per global crop in the batch, a center c of shape (K_p,), and a temperature. It returns $\operatorname{softmax}((\mathbf{z}_t - \mathbf{c})/\tau_t)$ , one probability vector per row. Subtract the center first and divide by the temperature second, because the center is a running mean of raw logits. Keep it numerically stable for logits in the thousands at $\tau_t = 0.04$ .

update_center(c, z_t, m_c) returns the new center $m_c\mathbf{c} + (1 - m_c)\,\bar{\mathbf{z}}_t$ , where $\bar{\mathbf{z}}_t$ is the mean of the rows of z_t. In a real step the targets of the current batch are computed first and the center is updated afterwards, the order of the official code.

import numpy as np

def teacher_probs(z_t, c, tau_t):
    raise NotImplementedError

def update_center(c, z_t, m_c):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

A tiny DINO you can run

The theorems describe the teacher's output. To see the whole loop, train one. The cell below builds 600 points in $\R^8$ from 4 hidden clusters. All of them share an offset of 2 in every coordinate, a stand-in for what all natural images have in common. The network never sees the cluster labels.

The student is a one-layer network $\mathbf{h} = \tanh(\mathbf{x}\mathbf{A} + \mathbf{b})$ followed by a DINO head of 16 unit-length prototypes, so each logit is a cosine. Two views of a point are the point plus independent noise. The teacher is an EMA with $m = 0.99$ . The gradient is the boxed $(\mathbf{p}_s - \mathbf{p}_t)/\tau_s$ , pushed back through the normalization by hand. Centering and sharpening can each be switched off.

Python · cell 2

Python · cell 3

The picture matches the paper's collapse study, with one difference. In the run without sharpening, the KL is still about 0.8 when the run ends, not zero. The network now gives every point nearly the same feature. The teacher's target is uniform only because its center subtracts the logits of that shared feature. The student has no center. To output a uniform distribution, it must turn its 16 prototypes until all of them make the same angle with that one feature, and in this toy that is slow. The KL is still falling, to about 0.70 after 2,000 steps and 0.64 after 6,000. What matters is that the target no longer depends on the input. In real DINO, both failed runs reach a KL of zero.

Multi-crop, small views that must explain the whole

DINO's third ingredient is to train on many crops of each image at two sizes. The idea was introduced with SwAV (Caron et al. 2020), and DINO makes it central. Crops are drawn by RandomResizedCrop, which picks a random rectangle covering a random fraction of the image's area and resizes it to a fixed size.

For a ViT with patch side $P = 16$ , a $96\times96$ crop is a $6\times6$ grid of patches. The position embeddings were learned for the $14\times14$ grid of a $224\times224$ input. They are resized to $6\times6$ by bicubic interpolation, as in §27.1, so one network handles both sizes.

Why does this help so much? Look at what one loss term asks. The teacher sees most of the photo and summarizes it. The student sees a patch covering perhaps 5 percent of it, say a wheel, and must produce the same summary. It can do that only by recognizing the wheel as part of a car. These local-to-global correspondences are a much harder prediction than matching two large, overlapping crops. In the paper's component ablation (Table 7) they are worth about 5 points of k-NN accuracy, 72.8 percent with multi-crop against 67.9 without.

The cost grows more slowly than the number of crops, for two reasons. The teacher never processes a local crop. And the attention cost of a crop grows with the square of its token count, so small crops are cheap to attend over. The cell counts it for a ViT-S/16, using the forward-pass cost of a ViT from §27.2, $L(24Td^2 + 4T^2d)$ FLOPs for $L$ blocks of width $d$ on $T$ tokens.

Python · cell 4

In practice, multi-crop buys more accuracy per GPU-hour than longer training does, so it is the ingredient to keep when compute is tight. An interviewer will often ask you to count its loss terms and its cost, as the exercise below does.

🔬 The scale split, the term count, and where multi-crop fails

The scale split. With global crops drawn from $(s, 1)$ and local crops from $(0.05, s)$ , k-NN accuracy in a short ViT-S run was 65.6, 68.0, 69.7, 69.8, and 69.5 percent for $s$ of 0.08, 0.16, 0.24, 0.32, and 0.48 (Appendix E). A plausible reading is that too small an $s$ lets global crops be nearly as small and uninformative as local ones. Too large an $s$ makes the two global crops overlap so much that matching them teaches little, though the drop at 0.48 is small. The released code uses $s = 0.4$ .

The term count. For $V_g = 2$ and $V_l = 10$ there are $2\times11 = 22$ loss terms. The official implementation (the DINOLoss class in main_dino.py) loops over the two teacher crops and all student crops and skips the pair where both are the same global crop. It divides the sum of the 22 batch-averaged cross-entropies by the number of terms, which is the $1/(V_g(V - 1))$ in the definition.

Where it fails. Multi-crop is not a free win for every loss. The paper reports that adding it to BYOL did not work out of the box (Table 14 and Appendix E), so its benefit depends on the loss it is paired with.

The widget draws the crops on one scene and counts their cost for a ViT-S/16 as you change the number of local crops, their size, and the scale split.

Interview

The DINO multi-crop loss and its gradient

Implement dino_loss(student, teacher, c, tau_s, tau_t), the full DINO loss with its gradient.

student is a list of $V = V_g + V_l$ arrays of shape (B, K_p), the student logits of every crop, with the $V_g$ global crops first.
teacher is a list of $V_g$ arrays of the same shape, the teacher logits of the global crops in the same order.

Return (loss, grads). The loss is the cross-entropy $H(\mathbf{p}_t, \mathbf{p}_s)$ averaged over the batch and over the $V_g(V - 1)$ pairs. Skip each pair of a teacher crop with the student's view of the same crop. The grads are a list of $V$ arrays, the gradient of loss with respect to each student array. The teacher is centered and sharpened, the student only sharpened, and the teacher receives no gradient. Your code must stay finite for logits of size 50 at $\tau_s = 0.1$ , and must not loop over the batch.

import numpy as np

def dino_loss(student, teacher, c, tau_s, tau_t):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

The real way in PyTorch

The production version has two parts. A loss module owns the center as a registered buffer, which is saved and loaded with the model but never trained, and it owns the teacher's temperature schedule. The teacher is a deep copy of the student with gradients turned off.

The block below runs five steps on random token tensors with a stand-in backbone, so that global and local crops have different numbers of tokens, as in a ViT. It also checks that the teacher never receives a gradient. With several GPUs, the batch mean in update_center is first averaged across devices by an all-reduce (§10.2), so that every GPU applies the same center.

🔥 In PyTorchfor a machine with PyTorch

What the features learn

DINO's features are judged frozen. The backbone stays fixed, and only a cheap readout is trained, or none at all. §28.6 treats these protocols in depth. The linear probe is the standard readout, and two others made DINO famous.

The classifier has no trained parameters and no settings beyond $k$ and the temperature 0.07. So it measures how well the raw geometry of the features already separates classes. A k-NN accuracy close to the linear-probe accuracy means the classes are already clustered, with no learned readout needed.

The second readout is the attention map. Take the last layer of a DINO ViT and look at the attention weights from the class token to the patch tokens, one map per head. Without any segmentation training, the maps light up on the foreground objects, with different heads on different objects or parts.

Caron et al. turned this into a number. For each head, keep the smallest set of patches that holds 60 percent of the class token's attention mass. Compare that mask with the ground-truth segmentation of the PASCAL VOC benchmark by the Jaccard index, the area of the overlap divided by the area of the union (the IoU of §27.6). Report the best head.

Why would it happen? A label like "dog" can be predicted from a few telling patches, such as an ear or a nose, so a supervised network learns to attend to those and ignore the rest. A self-supervised target has to describe a whole crop well enough to match another view of the same image. That rewards attending to everything that belongs to the main object. The attention-distance analysis of §27.3 shows the same thing from the other side, ViTs learning from data alone which patches go together.

These features became the starting point for unsupervised dense methods. LOST (Siméoni et al. 2021) finds the main object in an image from the similarity of the last layer's patch keys. It seeds at the patch with the fewest positively correlated patches, since background patches resemble many others. STEGO (Hamilton et al. 2022) distills the correspondences between DINO patch features, within and across images, into a small segmentation head, giving semantic segmentation without labels. And DINO features track objects through video with no training for the task. On the DAVIS video benchmark, each frame's patches are matched to the previous frame's by nearest neighbors.

In practice, these results are why DINO-family features are the default input to unsupervised segmentation and correspondence pipelines. An interviewer may ask why self-supervised ViTs segment and supervised ones do not, and the answer is the whole-crop argument above.

Core

Attention maps to masks

Turn attention maps into masks and score them, as the paper did.

attention_to_mask(attn, keep=0.6) takes attn of shape (heads, H_p, W_p), the class token's attention over the patch grid for each head. The register tokens and the class token itself are already removed (§27.4). Normalize each head's map to sum to 1. Then, for each head, keep the smallest set of patches whose total attention is at least keep, taking patches in order of decreasing attention. Here 0 < keep <= 1. Return a boolean array of the same shape.

Then implement jaccard(mask, gt) for two boolean arrays of the same shape, $|M \cap G|/|M \cup G|$ , defined as 1.0 when both are empty. The tests use maps without ties.

import numpy as np

def attention_to_mask(attn, keep=0.6):
    raise NotImplementedError

def jaccard(mask, gt):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

DINOv2, the same idea with curated data and scale

DINO was trained on ImageNet-1k, photographs chosen by people for a classification benchmark. A general-purpose encoder needs far more images. The natural source, raw web images, is badly unbalanced, with millions of near-identical product shots and almost nothing of rare subjects. DINOv2 (Oquab et al. 2023) kept the self-distillation core and made three changes. It added a loss on individual patches, added a regularizer that spreads the features apart, and spent much of its effort on data.

The patch loss applies DINO's loss to single patches instead of whole images. Some patches of the student's global crop are hidden behind a learned mask token, one vector that replaces each hidden patch. For each hidden patch, the student must predict the teacher's distribution for that patch, which the teacher computes from the unhidden crop. This loss was introduced as iBOT (Zhou et al. 2022), and it is what gives the patch features their quality. §28.5 defines it in full, next to the other masked image models.

📐 DINOv2

The loss is a weighted sum of three terms, $\mathcal{L} = \mathcal{L}_{\text{DINO}} + \mathcal{L}_{\text{iBOT}} + 0.1\,\mathcal{L}_{\text{KoLeo}}$ .

$\mathcal{L}_{\text{DINO}}$ is the image-level loss above, on the class token.
$\mathcal{L}_{\text{iBOT}}$ is the same cross-entropy applied to patches. Some patches of the student's global crops are replaced by a mask token. For each masked patch, the student's output is trained to match the teacher's output for that patch, which the teacher sees unmasked. §28.5 treats this as masked image modeling with a tokenizer that is learned online.
$\mathcal{L}_{\text{KoLeo}}$ spreads the class tokens of a batch apart (defined below).

The image-level and patch-level losses use separate heads, each with its own 128k prototypes in the ViT-g. The teacher's targets are balanced by 3 iterations of Sinkhorn-Knopp, the balancing step of SwAV (§28.3), instead of the running center. The architecture is a ViT with patch side 14 and SwiGLU feed-forward layers, the gated layers of §5.3. The largest is a ViT-g/14 with 1.1 billion parameters (40 blocks, width 1536, 24 heads), wider than the standard ViT-g/14 of §27.2, which has width 1,408 and about 1.0 billion. The teacher momentum follows a cosine from 0.994 to 1, and a short phase at the end trains at $518\times518$ pixels. Smaller models (ViT-S, B, L) are distilled from the frozen ViT-g rather than trained from scratch.

The Sinkhorn-Knopp swap is a stronger version of centering. Centering removes one shared vector using a running mean. Sinkhorn-Knopp rescales the batch's teacher scores so that every prototype receives the same total mass across the batch, while each image's target still sums to one.

KoLeo, a push away from the nearest neighbor

Contrastive methods spread features apart with negatives. DINO spreads its targets with centering, but nothing pushes the class tokens themselves to fill the space. Features that cluster too tightly retrieve badly, because every query then has many almost equally close neighbors. DINOv2 adds a regularizer that pushes every feature away from its nearest neighbor in the batch (Sablayrolles et al. 2019).

In plain words, the loss is large when some point sits very close to another, and it falls as every point gets room of its own. Take three unit vectors, two of them 0.1 apart and the third at distance 1 from both. Their nearest-neighbor distances are 0.1, 0.1, and 1, so the loss is $-\frac13(2\ln 0.1 + \ln 1) = 1.54$ . Spread the three to a mutual distance of 1 and it falls to 0. It can go lower still, since the loss is negative once every distance exceeds 1. Three unit vectors are farthest apart at 120 degrees on a circle, a mutual distance of $\sqrt3$ , where the loss is $-\ln\sqrt3 = -0.55$ .

To see how the regularizer acts during training, look at its gradient. It is a repulsion between each point and its nearest neighbor, with a force that grows as the two approach.

∴ The KoLeo gradient pushes each point away from its nearest neighbor

Fix the nearest-neighbor assignment $j(i)$ , which is constant except where two neighbors tie. On the normalized points $\hat{\mathbf{h}}$ ,

\frac{\partial\mathcal{L}_{\text{KoLeo}}}{\partial\hat{\mathbf{h}}_k} = -\frac{1}{n}\,\frac{\hat{\mathbf{h}}_k - \hat{\mathbf{h}}_{j(k)}}{d_k^2} \;+\; \frac{1}{n}\sum_{i\,:\,j(i) = k}\frac{\hat{\mathbf{h}}_i - \hat{\mathbf{h}}_k}{d_i^2}.

A gradient step therefore moves $\hat{\mathbf{h}}_k$ directly away from its nearest neighbor, with a force of magnitude $1/(n d_k)$ , and away from every point that has $k$ as its nearest neighbor. Through the normalization, $\partial\mathcal{L}/\partial\mathbf{h}_k = (\mathbf{I} - \hat{\mathbf{h}}_k\hat{\mathbf{h}}_k^\top)\,(\partial\mathcal{L}/\partial\hat{\mathbf{h}}_k)/\lVert\mathbf{h}_k\rVert$ , the component tangent to the sphere.

∂ Proof of the KoLeo gradient

$\nabla_{\mathbf{u}}\log\lVert\mathbf{u} - \mathbf{v}\rVert = (\mathbf{u} - \mathbf{v})/\lVert\mathbf{u} - \mathbf{v}\rVert^2$ . The point $\hat{\mathbf{h}}_k$ appears in its own term (as $\mathbf{u}$ ) and in the term of every $i$ with $j(i) = k$ (as $\mathbf{v}$ , with the opposite sign), and the overall factor is $-1/n$ . The magnitude of the first term is $\lVert\hat{\mathbf{h}}_k - \hat{\mathbf{h}}_{j(k)}\rVert/(n d_k^2) = 1/(n d_k)$ . For the normalization, the Jacobian of $\mathbf{h}\mapsto\mathbf{h}/\lVert\mathbf{h}\rVert$ is $(\mathbf{I} - \hat{\mathbf{h}}\hat{\mathbf{h}}^\top)/\lVert\mathbf{h}\rVert$ , which is symmetric. ∎

In plain words, the closer two features are, the harder KoLeo pushes them apart, and the force grows without bound as they meet. So near-duplicates in a batch are separated first. The logarithm makes the loss independent of scale, and the normalization stops it from being satisfied by inflating norms. DINOv2 applies it with weight 0.1 to the student's class tokens of each GPU's share of the batch. The paper's appendix describes the first global crop only. The released code applies it to each of the two global crops separately, so that two crops of the same image are never pushed apart. The cell compares a tightly bunched batch with a spread-out one.

Python · cell 5

The bunched batch has a mean pairwise cosine of 0.98 and a KoLeo loss of 2.10, more than ten times the spread batch's 0.18. Tightly bunched features are exactly what the regularizer penalizes, and they are also what makes nearest-neighbor retrieval fail. That is why removing KoLeo hurt DINOv2's retrieval most, as the ablation below shows.

Interview

KoLeo and its gradient

Implement koleo(H, eps=1e-8), the regularizer and its gradient. H has shape (n, d).

Normalize the rows. Then find each row's nearest other row by Euclidean distance on the normalized rows, never the row itself. Return (loss, grad), where loss is $-\frac1n\sum_i\log(d_i + \epsilon)$ and grad is its gradient with respect to the unnormalized H, of shape (n, d). Do not loop over pairs. The tests use data without nearest-neighbor ties.

import numpy as np

def koleo(H, eps=1e-8):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Curating a dataset without labels

DINOv2's data pipeline uses no text, no metadata, and no labels. It uses only image similarity, measured with the features of a self-supervised model.

📌 LVD-142M

DINOv2 started from a crawl of web images. It removed unsafe URLs, deduplicated with PCA hashing, filtered adult content, and blurred faces, leaving 1.2 billion unique images. It then removed near-duplicates with a copy-detection model, both within the pool and against the test sets of every benchmark it reports, the image version of the deduplication in §8.3. Finally it embedded everything with a self-supervised ViT-H/16 and used a set of curated datasets as queries (ImageNet-22k, ImageNet-1k, Google Landmarks, and fine-grained sets). For each query image it retrieved nearest neighbors from the pool by cosine similarity, typically 4 per query. When the curated set was small, it instead sampled from the query's k-means cluster. The result is 142 million images. The search used FAISS with inverted-file, product-quantized GPU indices (§7.4) and took under two days on 20 nodes of 8 V100 GPUs.

Curation is not a uniform win, and the ablation shows where it pays. The benchmarks need a word first. ImageNet-A is a set of natural photographs that standard ImageNet classifiers get wrong. iNaturalist 2021 is fine-grained species classification. Oxford-M is the medium split of a landmark retrieval benchmark, scored in mean average precision (mAP). ADE20k is semantic segmentation of scenes into 150 classes, scored in mIoU, the overlap over union of predicted and true regions, averaged over classes.

A ViT-g was trained for the same number of iterations on LVD-142M and on 142 million random images from the same pool (Oquab et al. 2023, Table 2). The curated data won by 85.8 against 83.3 percent on ImageNet-1k, 73.9 against 59.4 on ImageNet-A, 64.6 against 54.3 mAP on Oxford-M, and 86.4 against 76.4 on iNaturalist 2021. But it lost on ADE20k segmentation, 47.7 against 48.5 mIoU.

Retrieval toward curated sets pulls the data toward what those sets contain, objects and fine-grained categories. That helps recognition a lot and dense scene understanding not at all. DINOv3's data ablation found the same tension, with no single curation method best on every benchmark, so it mixes curation by clustering with curation by retrieval.

Most steps add a point or less, and together they add 9.1 points of k-NN accuracy. Scaled to 1.1 billion parameters on the curated data, the full recipe reaches the level of encoders trained on billions of captioned images, which CLIP and its successors build.

DINOv2's distillation reuses the training loop with a few changes. The teacher is the frozen ViT-g instead of an EMA. Masking and stochastic depth are removed, the iBOT loss is applied on both global crops, and the model kept at the end is an EMA of the student. This is the soft-label distillation of §20.4, with DINO's prototype distributions in place of class probabilities. It beats training the small model from scratch on all 12 of the paper's classification benchmarks. In practice this is why you almost never train a small self-supervised encoder from scratch. You distill it from a big one.

DINOv3, a 7B teacher and Gram anchoring

DINOv3 (Siméoni et al. 2025) set out to train the DINOv2 recipe at a much larger scale, for as long as it kept improving. Two problems had to be solved on the way. The data had to be balanced across everything the web shows, and the patch features had to be kept from decaying during long training.

📐 DINOv3

The teacher is a custom ViT-7B with 6.7 billion parameters, 40 blocks, width 4096, 32 heads of width 128, a SwiGLU feed-forward layer of hidden width 8192, and patch side 16. It replaces learned position embeddings with axial RoPE, rotary position embeddings applied separately along rows and columns (§27.1). The coordinate box is randomly rescaled during training for robustness to resolution and aspect ratio. It has 4 register tokens, extra learned tokens that absorb global computation so the patch tokens stay clean (§27.4). Its DINO head is an MLP of widths 8192, 8192, 512 with 256k prototypes, and its iBOT head has 96k prototypes.

The pretraining loss is DINOv2's, $\mathcal{L}_{\text{DINO}} + \mathcal{L}_{\text{iBOT}} + 0.1\,\mathcal{L}_{\text{KoLeo}}$ , with Sinkhorn-Knopp in both terms and KoLeo computed on small groups of 16 class tokens. Learning rate, weight decay, and teacher momentum are constant (learning rate 0.0004, weight decay 0.04, momentum 0.999). Only the learning rate and the teacher temperature warm up linearly at the start, the learning rate over 100k iterations. The batch is 4,096 images on 256 GPUs, each image with 2 global crops of 256 pixels and 8 local crops of 112 pixels, for 1 million iterations. Three phases follow, Gram anchoring, a high-resolution phase, and distillation into a family of smaller models.

Two choices in the box deserve a sentence each. A patch side of 16 at 256 pixels gives a 16 by 16 grid, the same 256 patches as DINOv2's patch 14 at 224 pixels. So the sequence length per global crop is unchanged while the images are larger. And the constant schedules break with DINO and DINOv2, whose cosine schedules fix the length of training in advance. At this scale nobody can guess the right length, so DINOv3 keeps every schedule constant and decides when to stop by watching the evaluations.

Balanced data by hierarchical k-means

Why hierarchical? Plain k-means on web images does not balance them. The k-means objective places more centroids where the data is dense, because splitting a crowded region lowers the squared error more than covering a sparse one. So a million cat photos get many clusters, and a rare species gets none of its own. Taking the same number of images from each cluster then still over-represents what was common.

Running k-means again on the centroids fixes this. The first level's centroids are spread more evenly than the images, though still crowded where the images are. Clustering those centroids spreads them more evenly still. After a few levels, the top level has roughly one cluster per broad concept, however many images each concept had.

🔬 Why each level flattens the distribution

Vo et al. (2024) made the argument quantitative with a classical result of quantization theory (Zador 1982). With many centroids in $D$ dimensions, k-means centroids are distributed with density proportional to $p^{D/(D+2)}$ , where $p$ is the data density. That is flatter than $p$ but far from flat, and the larger $D$ is, the closer the exponent is to 1. Running k-means again on the centroids applies the exponent again, giving density $p^{(D/(D+2))^T}$ after $T$ levels, which tends to uniform as $T$ grows.

The formula is asymptotic. For real embeddings the $D$ that matters is the dimension in which the data actually varies, far below the embedding width. So read it as the mechanism, not as a count of how many levels you need. Each level has far fewer points than the one below. So the full method also re-runs each level's k-means on the few points nearest each centroid, which are spread more evenly, and then reassigns all points to the new centroids.

Sampling runs top-down. Given a budget for a cluster, split it among the cluster's children so that small children are taken whole and the large ones share what remains equally, then recurse to the leaves. This rule is called water-filling, because it fills every child up to a common level, as water fills the uneven bottom of a tank. For a budget $N$ and child sizes $s_1, \dots, s_k$ , give child $j$ the amount $\min(t, s_j)$ for the largest level $t$ that keeps $\sum_j\min(t, s_j) \le N$ . Then hand out any remainder one unit at a time to the children not yet full.

Research

Balanced data by hierarchical k-means

Build a small version of DINOv3's data curation and measure what it buys. The setup defines kmeans(X, k, seed), which returns (centroids, labels) by Lloyd's algorithm with k-means++ seeding (§7.4). It also defines a long-tailed dataset X_lt of 8-dimensional embeddings. Their true concepts concept_lt (40 of them, from 2,400 points down to about 20) may be used only for evaluation. Implement four functions.

allocate(sizes, n), water-filling. Return integer counts a with a[j] <= sizes[j] and sum(a) == n (assume n <= sum(sizes)). Set a[j] = min(t, sizes[j]) for the largest integer t with $\sum_j\min(t, s_j) \le n$ . Then give the remaining units, one each, to the clusters with sizes[j] > t in order of increasing index.
two_level_kmeans(X, k_fine, k_coarse). Run kmeans(X, k_fine, seed=0) on the points, then kmeans on the fine centroids with k_coarse and seed=1. Return (fine, parent), the fine label of every point and the coarse label of every fine cluster.
balanced_sample(fine, parent, n, seed=0). Allocate n across coarse clusters by their sizes in points, then split each coarse cluster's share across its fine clusters, then draw that many distinct points uniformly from each fine cluster with np.random.default_rng(seed). Return the sorted array of point indices.
coverage(labels, n_concepts). Return (covered, entropy), the number of distinct concepts present in labels and the entropy in nats of their histogram.

import numpy as np

def allocate(sizes, n):
    sizes = np.asarray(sizes)
    return np.round(n * sizes / sizes.sum()).astype(int)     # proportional: uniform sampling in disguise

def two_level_kmeans(X, k_fine, k_coarse):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

On this data the hierarchical sample of 400 points covers all 40 concepts, with a histogram entropy of 3.62 nats, close to the maximum $\ln 40 = 3.69$ . A uniform sample of the same size covers 37 concepts with entropy 2.66, dominated by the most common concepts. Balancing over the 60 first-level clusters alone gives about 3.4, because k-means spent 9 of its 60 clusters on the largest concept. That is exactly the density bias the second level removes. Vo et al. found that every added level gave a better balanced dataset and, mostly, better features, with the biggest jump from one level to two. DINOv3 used five.

Why dense features decay, and Gram anchoring

Train the DINOv2 recipe at 7 billion parameters for a million iterations and watch two curves (Siméoni et al. 2025, Figure 5). ImageNet linear accuracy on the class token rises the whole time. Linear segmentation on PASCAL VOC from the patch features peaks around 200k iterations and then falls, for the ViT-7B below where it started.

The patch features show why. At 200k iterations, the cosine similarity between one patch and all others is smooth and concentrated on the object the patch belongs to. By 600k iterations many unrelated patches have become similar to it, and the cosine between the class token and the patches has grown steadily. The global objective is slowly making every patch look like the image summary. These are not the high-norm artifacts that registers fixed (§27.4), since with registers the patch norms stay stable throughout. It is a loss of locality.

The fix starts from an observation. Global quality and dense quality are nearly independent, so it should be possible to protect the second without fighting the first. Gram anchoring constrains only which patches are similar to which. Its name comes from the Gram matrix $\mathbf{X}\mathbf{X}^\top$ of a set of feature vectors stored as the rows of $\mathbf{X}$ , the table of all their pairwise dot products (§0.10). For unit vectors, those dot products are cosine similarities.

📐 Gram anchoring

For one global crop with $N_p$ patches, let $\mathbf{X}_S \in \R^{N_p\times d}$ hold the student's $\ell_2$ -normalized patch features. Let $\mathbf{X}_G \in \R^{N_p\times d}$ hold those of a Gram teacher, an earlier checkpoint of the EMA teacher whose dense features were still good. The Gram loss is

\mathcal{L}_{\text{Gram}} = \big\lVert \mathbf{X}_S\mathbf{X}_S^\top - \mathbf{X}_G\mathbf{X}_G^\top\big\rVert_F^2 ,

the squared Frobenius distance (the sum of squared entry differences) between the two matrices of patch-to-patch cosine similarities. It is computed on global crops only. DINOv3 adds it with weight 2 in a refinement phase that starts after 1 million iterations. The Gram teacher is refreshed to the current EMA teacher every 10k iterations, at most three times. Its features are computed at twice the resolution and downsampled by 2 with bicubic interpolation, which makes them smoother.

The loss pins down similarities and nothing else. Rotating every patch feature by the same rotation leaves all cosines unchanged, so the loss cannot see it. The theorem says that this is the only freedom it leaves. Here an orthogonal matrix $\mathbf{R}$ is one with $\mathbf{R}\mathbf{R}^\top = \mathbf{I}$ , a rotation possibly combined with a reflection.

Proof of 1. $(\mathbf{X}\mathbf{R})(\mathbf{X}\mathbf{R})^\top = \mathbf{X}\mathbf{R}\mathbf{R}^\top\mathbf{X}^\top = \mathbf{X}\mathbf{X}^\top$ , since $\mathbf{R}\mathbf{R}^\top = \mathbf{I}$ . ∎

∂ Proof of part 2

Let $\mathbf{G} = \mathbf{X}\mathbf{X}^\top = \mathbf{Y}\mathbf{Y}^\top$ have rank $r$ and eigendecomposition $\mathbf{G} = \mathbf{U}\boldsymbol\Lambda\mathbf{U}^\top$ with $\mathbf{U} \in \R^{N_p\times r}$ orthonormal and $\boldsymbol\Lambda$ positive diagonal (§0.10). Define $\mathbf{A} = \mathbf{X}^\top\mathbf{U}\boldsymbol\Lambda^{-1/2} \in \R^{d\times r}$ . Its columns are orthonormal, because $\mathbf{A}^\top\mathbf{A} = \boldsymbol\Lambda^{-1/2}\mathbf{U}^\top\mathbf{G}\mathbf{U}\boldsymbol\Lambda^{-1/2} = \mathbf{I}$ . The column space of $\mathbf{X}$ equals that of $\mathbf{G}$ , which is spanned by $\mathbf{U}$ , so $\mathbf{X} = \mathbf{U}\mathbf{U}^\top\mathbf{X} = \mathbf{U}\boldsymbol\Lambda^{1/2}\mathbf{A}^\top$ . The same construction gives $\mathbf{Y} = \mathbf{U}\boldsymbol\Lambda^{1/2}\mathbf{A}'^\top$ with orthonormal $\mathbf{A}'$ . Complete both to orthogonal matrices $\mathbf{Q} = [\mathbf{A}, \mathbf{A}_\perp]$ and $\mathbf{Q}' = [\mathbf{A}', \mathbf{A}'_\perp]$ and set $\mathbf{R} = \mathbf{Q}\mathbf{Q}'^\top$ , which is orthogonal. Then $\mathbf{X}\mathbf{R} = \mathbf{U}\boldsymbol\Lambda^{1/2}\mathbf{A}^\top[\mathbf{A}, \mathbf{A}_\perp]\mathbf{Q}'^\top = \mathbf{U}\boldsymbol\Lambda^{1/2}[\mathbf{I}, \mathbf{0}]\,\mathbf{Q}'^\top = \mathbf{U}\boldsymbol\Lambda^{1/2}\mathbf{A}'^\top = \mathbf{Y}$ . ∎

In plain words, the Gram teacher says "these two patches are alike, those two are not", and the student must keep saying the same. But it may express each patch in whatever coordinates the global objective prefers. That is why the loss can repair the dense features without dragging the global ones back to an earlier, weaker state. The cell checks both halves of that claim on 49 random patch features.

Python · cell 6

The rotated features are 10.3 away from the teacher's as vectors, yet their Gram loss is at rounding error. The perturbed features are closer as vectors, 6.9 away, but their similarities have changed and they pay a Gram loss of 114. That asymmetry is the whole design.

An interviewer who asks how to fix decaying dense features expects this answer. Anchor the similarity structure to an earlier checkpoint, and leave the features themselves free.

What DINOv3 delivers

Distillation finishes the pipeline. The 7B model, now frozen rather than an EMA, teaches ViT-S (22M parameters), a custom ViT-S+ (29M), ViT-B (86M), ViT-L (0.3B), a custom ViT-H+ (0.8B), and ConvNeXt models from Tiny to Large, all with the same training objective. ConvNeXt is a convolutional network modernized with Transformer-era design choices (§27.5).

The benchmarks below are standard. ADE20k (150 scene classes), Cityscapes (street scenes), and VOC are segmentation sets scored in mIoU. NYUv2 (indoor) and KITTI (driving) are depth sets scored by RMSE, the root mean squared error of the predicted depth, where lower is better. COCO detection is scored in mAP, the mean average precision over classes.

The table puts DINOv3 next to the strongest frozen encoders of other kinds. SigLIP 2 and PEcore, the core model of Perception Encoder, are image-text encoders. They train an image encoder and a text encoder so that a photo and its caption get similar vectors, the approach of CLIP. SigLIP scores each image-caption pair with its own yes-or-no sigmoid instead of a softmax over the batch (§30.4). AM-RADIO is distilled from several teachers at once. All numbers come from the same two tables of the DINOv3 paper, evaluated by its authors with one protocol. Higher is better except for depth error.

backbone	how it was trained	ImageNet linear	ADE20k	Cityscapes	VOC	NYUv2 RMSE
DINOv3 7B/16	self-distillation, curated web images	88.4	55.9	81.1	86.6	0.309
DINOv2 g/14	self-distillation, LVD-142M	87.3	49.5	75.6	83.1	0.372
Web-DINO 7B/14	DINOv2 recipe on 2B MetaCLIP web images	85.9	42.7	68.3	76.1	0.466
Franca g/14	open data and code	84.8	46.3	68.7	82.9	0.445
SigLIP 2 g/16	image-text contrastive	89.1	42.7	64.8	72.7	0.494
PEcore G/14	image-text contrastive	89.3	38.9	61.1	69.2	0.590
AM-RADIOv2.5 g/14	distilled from DINOv2, CLIP, SAM	88.0	53.0	78.4	85.4	0.340

The pattern is the one this chapter predicts. Encoders trained to match captions are slightly better at naming the whole image and much worse at describing its parts, because a caption summarizes the whole image. Self-distillation with a patch loss and Gram anchoring wins every dense column. Its margin over the text-trained encoders is wide, 13 mIoU or more on ADE20k, while it stays within a point of them on classification. Only AM-RADIOv2.5, which is itself distilled partly from DINOv2 and from the segmentation model SAM, comes within 3 points.

Two rows are instructive. Web-DINO (Fan et al. 2025) scales the DINOv2 recipe to 7B parameters on 2 billion web images from the MetaCLIP collection, without hierarchical curation or Gram anchoring. It trails DINOv2 by 1.4 points on ImageNet and by 6.8 mIoU on ADE20k, so scale alone does not make good dense features. Franca (Venkataramanan et al. 2025) is the fully open attempt at the recipe, since LVD-142M and LVD-1689M are private, and the gap in its row is partly a data gap.

Using one of these models takes a few lines, with one trap. Register tokens sit between the class token and the patches. They must be dropped before the patch features are reshaped into a grid, or every dense map is shifted (the pitfall of §27.4).

Choosing among the generations

	DINO	DINOv2	DINOv3
released	April 2021	April 2023	August 2025
data	ImageNet-1k, 1.28M	LVD-142M, retrieval toward curated sets	LVD-1689M, hierarchical k-means plus retrieval plus public sets
largest model	ViT-B/8, 86M	ViT-g/14, 1.1B	ViT-7B/16, 6.7B
losses	DINO	DINO, iBOT, KoLeo	DINO, iBOT, KoLeo, then Gram anchoring
collapse control	centering, sharpening, momentum	Sinkhorn-Knopp, sharpening, momentum	Sinkhorn-Knopp, sharpening, momentum
schedules	cosine (lr, wd, momentum)	cosine	constant after warmup
positions	learned, interpolated	learned, interpolated	axial RoPE, 4 registers
ImageNet frozen	80.1 linear, 77.4 k-NN	86.5 linear, 83.5 k-NN	88.4 linear
pretraining compute	two 8-GPU servers for 3 days (ViT-S/16)	22,016 A100 GPU-hours	61,440 H100 GPU-hours
use it when	you must train your own model on modest data and compute	you want a strong, widely supported off-the-shelf encoder	you need the best frozen dense features, at any resolution

For a new project the choice is almost always a distilled DINOv2 or DINOv3 student of the size you can afford, frozen or lightly fine-tuned. Run the evaluation protocols of §28.6 on your own data before committing. Training DINO yourself still makes sense on a domain far from web photographs, such as medical slides, satellite images, or microscopy, where curated pools do not exist. DINOv3 trained a separate model on 493 million satellite images for that reason.

These features have spread far beyond classification. Vision-language models fuse DINOv2 features with those of a text-trained encoder to recover the fine geometry that captions miss (connecting a vision encoder to a language model). Image generators train faster when the internal features of a diffusion transformer are aligned with DINOv2's (diffusion transformers), and evaluators of generated images now often use DINOv2 features in place of the older Inception network. Speech models adopted the EMA teacher that supplies the student's targets (data2vec and its relatives). And a moving-average teacher that supplies targets in feature space is also the core of joint-embedding predictive video models, which predict missing parts of a video in feature space rather than in pixels.

⚠️ Five ways to get DINO wrong

Centering after the softmax, or dividing by $\tau_t$ before subtracting a center computed on raw logits. Both change what the center removes. The second subtracts $\mathbf{c}$ from logits already divided by $\tau_t$ , so it removes only a fraction $\tau_t$ of the shared part, 4 percent at $\tau_t = 0.04$ .
A teacher temperature that is too high from the start. Above about 0.06 with $\tau_s = 0.1$ the run collapses to the uniform target, with the loss stuck at $\ln K_p$ . Warm it up from 0.04.
A center that moves too slowly, or a teacher that moves too fast. $m_c = 0.999$ collapsed in the paper, while $m_c = 0$ worked. A teacher equal to the student (no momentum) collapsed too.
Reading collapse from the loss. A collapsed run has a perfectly low KL. Monitor the teacher's entropy and $I(\text{image}; \text{prototype})$ , or the k-NN accuracy, which drops to chance.
Forgetting the registers and the class token when reshaping patch features, or judging a long run only by ImageNet accuracy while its dense features decay.

🧭 What to remember

DINO trains a student to match a momentum teacher, $\xi \leftarrow m\xi + (1-m)\theta$ , with the loss $H(\operatorname{softmax}((\mathbf{z}_t - \mathbf{c})/\tau_t), \operatorname{softmax}(\mathbf{z}_s/\tau_s))$ and student gradient $(\mathbf{p}_s - \mathbf{p}_t)/\tau_s$ . The teacher, an average of the last $1/(1-m)$ students, is better than the student throughout training.
Collapse is any target that does not depend on the image. It shows as $I(\text{image}; \text{prototype}) = 0$ and never in the loss. Centering removes input-independent logits (a shared component survives with weight $m_c^n$ ). Sharpening, with $\tau_t$ below $\alpha\tau_s$ , makes image-specific differences grow. Each fails alone, toward one dimension or toward uniform.
Multi-crop pairs 2 teacher-seen global crops with all student crops, 22 terms for 10 local crops. It teaches local-to-global correspondence at 1.92 times the tokens but only 1.17 times the attention work, and gives more accuracy in half the time.
DINO ViT features give k-NN accuracy close to linear (74.5 against 77.0 for ViT-S/16) and class-token attention that segments objects (Jaccard 45.9 against 27.3 supervised), a property of self-supervised ViTs that labels suppress.
DINOv2 added a masked patch loss, Sinkhorn-Knopp, KoLeo (a nearest-neighbor push that maximizes an entropy estimate), and above all retrieval-curated data, reaching 86.5 percent linear with a 1.1B ViT-g and no text.
DINOv3, among the strongest self-supervised encoders published, trains a 7B teacher on 1.7 billion images balanced by hierarchical k-means, with constant schedules. It repairs the decay of dense features with Gram anchoring, a loss on patch similarities that ignores any rotation of the features. It leads every frozen dense benchmark (55.9 ADE20k linear mIoU) while staying within a point of text-trained encoders on ImageNet classification.
Every method so far compares summaries of whole crops, with DINOv2's patch loss as the first step beyond that. §28.5 asks what happens when the model must predict the missing parts of an image directly.

Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers. The DINO paper. Section 3 and Algorithm 1 define the method, Section 5 holds the teacher, collapse, and compute studies, and Appendices C to E the head, temperature, centering, and crop-scale ablations quoted here.
Oquab et al. (2023). DINOv2: Learning Robust Visual Features without Supervision. The data pipeline (Section 3), the combined loss (Section 4), and the ablations of Tables 1 to 3 are the best published account of what it takes to scale self-supervised vision.
Siméoni et al. (2025). DINOv3. The 7B teacher, LVD-1689M, the dense-feature decay and Gram anchoring (Section 4), multi-student distillation, and the frozen-backbone comparisons in Tables 3 and 7.
Tarvainen & Valpola (2017). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. The weight-averaged teacher that DINO generalizes to the case with no labels.
Zhou et al. (2022). iBOT: Image BERT Pre-Training with Online Tokenizer. The masked patch self-distillation loss that DINOv2 and DINOv3 add to DINO.
Sablayrolles et al. (2019). Spreading vectors for similarity search. Where the KoLeo regularizer comes from, derived for making features easy to index.
Vo et al. (2024). Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach. Why plain k-means does not balance web data, and the hierarchical k-means and balanced sampling that DINOv3 uses.
Venkataramanan et al. (2025). Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning. A fully open model and data pipeline in the DINOv2 family, the reference point for reproducing these results.

Made for larger screens

Self-distillation with DINO

A teacher built from the student

DINO, step by step

Two ways to collapse, and the two cures

What centering removes, exactly

Why sharpening is needed on top

Centering and sharpening

A tiny DINO you can run

Multi-crop, small views that must explain the whole

The DINO multi-crop loss and its gradient

The real way in PyTorch

What the features learn

Attention maps to masks

DINOv2, the same idea with curated data and scale

KoLeo, a push away from the nearest neighbor

KoLeo and its gradient

Curating a dataset without labels

DINOv3, a 7B teacher and Gram anchoring

Balanced data by hierarchical k-means

Balanced data by hierarchical k-means

Why dense features decay, and Gram anchoring

What DINOv3 delivers

Choosing among the generations

Read more

Watch next