How a vision model learns from millions of photographs that nobody has labeled. Pretext tasks, contrastive learning, the collapse problem and the methods that escape it, self-distillation from DINO to DINOv3, masked autoencoders, and how to measure what a representation knows, from first principles to the frontier, at interview depth.

Self-supervised learning trains an image encoder on photographs that nobody has labeled. An encoder is a network that turns an image into a vector of numbers, called its representation. Later tasks read that vector with a small classifier or a little extra training. With no labels, the training target has to come from the image itself. You hide or change part of a photo and ask the network about what you did, and to answer well it has to learn what is in the picture.

This matters because labels are the scarce resource in vision and pixels are not. Chapter 27 measured what that scarcity costs. A Vision Transformer trained on ImageNet's 1.28 million labeled photos alone loses to a convolutional network (§27.3). It pulls ahead with 300 million labeled images, and catches up on ImageNet alone only with a carefully tuned training recipe (§27.4). Self-supervision removes that dependence on labels.

The strongest image encoders trained without labels or text are self-supervised. DINOv3, a 7-billion-parameter encoder from 2025, learned from 1.7 billion unlabeled photos (Siméoni et al. 2025). On ImageNet its frozen features score within a point of the best encoders trained on images paired with captions. Frozen DINOv2 and DINOv3 features are now a default input for segmentation, depth, and robot perception. Masked-autoencoder pretraining, built in §28.5, starts models such as the image encoder of Segment Anything.

The chapter starts from a loss the book already has. The InfoNCE loss of §7.2, built there for text retrieval, scores each matching pair against many non-matching ones with a softmax. In this chapter the matching pair becomes two altered copies of one photo. This section explains why labels run out, how the first self-supervised methods worked, and how they cheated. Then it shows why the modern approach, which asks two such copies to get the same vector, learns the classes, with a theorem that says exactly when. It ends with the three tests that turn "a good representation" into a number, which every later section of the chapter uses.

Why labels are the bottleneck

Start with the sizes. ImageNet, in the version nearly every vision paper reports on (the 2012 challenge set, ILSVRC-2012), has 1.28 million training photographs. A person checked each one against its class, one of 1,000. That is the largest carefully labeled image collection most researchers ever use, and it is tiny next to what cameras produce.

The biggest supervised vision datasets gave up on careful labels long ago. JFT-300M, an internal Google dataset, has more than 375 million noisy labels on 300 million images (Sun et al. 2017). Mahajan et al. (2018) pretrained on up to 3.5 billion Instagram photos labeled only by their hashtags. Their best model reached 85.4 percent top-1 accuracy on ImageNet after fine-tuning. Top-1 accuracy is the fraction of test images whose highest-scoring class is the right one. Fine-tuning means training the whole pretrained network further on ImageNet's own labels.

Labels have three costs. They are expensive, since ImageNet's came from crowd workers who checked candidate images that search engines had returned. They are noisy at scale, since hashtags and automatic tags are mostly right but not always. And their vocabulary is fixed. A model trained on 1,000 classes has been told what matters. Anything those classes do not separate is, as far as the loss is concerned, noise to throw away.

That last cost has been measured. Kornblith et al. (2019) took 16 ImageNet classifiers and reused each one, frozen, on 12 other image datasets. For each dataset they kept the penultimate features, the vector just before the network's final classification layer, and trained only a new linear classifier on top. A linear classifier trained on frozen features is called a linear probe. Across the 16 networks, transfer accuracy tracked ImageNet accuracy closely, with a correlation of $r = 0.99$. But label smoothing and dropout, two regularizers that slightly improve ImageNet accuracy, gave penultimate features that transferred much worse. A representation tuned to separate 1,000 classes as cleanly as possible discards what those classes do not need.

A label can say only a few bits

The second problem is about information rather than cost. Recall from §0.5 that the entropy $H(Y)$ of a random variable $Y$ measures, in bits, how uncertain it is. The mutual information $I(X;Y) = H(Y) - H(Y\mid X)$ is how many of those bits you learn about $Y$ by seeing $X$. It is symmetric, so it is also what $Y$ tells you about $X$. Let $X$ be an image and $Y$ its label.

Proof. The conditional entropy of a discrete variable is never negative, so $I(X;Y) \le H(Y)$. Among distributions on $K$ values the uniform one has the largest entropy, $\log_2 K$. §0.5 proves this from the fact that the KL divergence to the uniform distribution $\mathcal{U}$ is never negative, $D_{\mathrm{KL}}(p\,\Vert\,\mathcal{U}) = \log K - H(p) \ge 0$. ∎

In plain words, however rich the image, its label can pass on no more than $\log_2 K$ bits about it. For ImageNet that is $\log_2 1000 \approx 9.97$, about 10 bits. The label "golden retriever" says nothing about the dog's pose, the light, the grass, or the frisbee.

The bound limits what a target can transmit. It is not a theory of learning. Supervised networks learn far richer features than 1.6 MB would suggest, because the images themselves shape what the network computes on its way to the label. The bound also says nothing about which bits are useful. A popular picture from a 2016 talk by Yann LeCun calls self-supervision the cake and labels the icing. It is a heuristic built on this calculation, not a theorem.

There are two ways around the bottleneck. One keeps labels but makes them free, by using the text that already sits next to images on the web, such as hashtags, captions, and alt-text. That is weak supervision, and it leads to the joint image and text spaces of CLIP and its successors. The other drops labels altogether and makes the images supervise themselves. That is self-supervised learning, and it is this chapter.

What this changes in practice is where the effort goes. When labels are the bottleneck, the work moves to choosing which unlabeled images to train on and which questions to ask of them. An interviewer will often open with "why not just label more data?" The answer names four limits. Labels cost money, they are noisy at scale, their vocabulary is fixed, and each one carries at most $\log_2 K$ bits.

Predicting one part of an image from the rest

Language solved this problem first. word2vec learned a vector for each word by predicting the words around it (§1.5), building on the observation of §1.4 that the company a word keeps carries its meaning. BERT learned by filling in words that had been hidden from it (§7.1). Neither needed a human label, because the text supplied both the question and the answer. Images need a different notion of "part", and the history of self-supervised vision is largely a search for the right one.

Here is the simplest image version. Rotate a photo by a random multiple of 90 degrees, and ask a network which multiple it was. The rotated photo is the question. The answer, 0, 1, 2, or 3, is known because you chose it. Nobody wants a rotation detector. But a network that can tell which way a photo was turned has had to find skies, faces, and the bottoms of objects.

That recipe is called a pretext task. It makes a question and its answer out of an unlabeled image, trains a network to answer, and then keeps only the part of the network that turned the image into a vector. The task is a pretext because nobody wants its answers. What we want is the encoder that solving it forces the network to build.

In plain words, corrupt or cut the image in a way you choose, and ask the network to undo or explain what you did. Every method in this chapter fits the template. In rotation prediction $c$ is an angle, $\tilde x$ the rotated image, and $y$ the angle's index. In masked modeling $\tilde x$ is the visible part of the image and $y$ the hidden part. In the joint-embedding methods of the later sections, $\tilde x$ is one random alteration of $x$, and the target is the network's own representation of a second alteration.

The target $y$ is a label the image made for itself, so the bound above applies to it too. Here is how many bits each classic target carries.

More bits do not mean better features. The jigsaw label carries as much as an ImageNet label, yet it learns much weaker features than the contrastive methods of the next section. What matters is that the target is hard to predict without understanding the image, and, as we will see, that no low-level accident predicts it instead. The cell checks the arithmetic.

A tour of the classic pretext tasks

Between 2014 and 2018 researchers proposed dozens of pretext tasks. Five of them shaped the field, and each taught a lesson that the modern methods still rely on.

Context prediction

Context prediction asks where one patch of a photo sits relative to another. Doersch, Gupta and Efros (2015) took word2vec's idea, predicting a word from its neighbors, and applied it to patches. Sample a 96 by 96 patch, then one of its 8 neighbors on a 3 by 3 grid, and ask a network which neighbor it is. To know that a patch of ear sits above a patch of eye, the network has to recognize ears and eyes.

Both patches go through the same convolutional tower, so the two branches share their weights. The tower is AlexNet, the 2012 ImageNet network of §0.8, up to its first fully connected layer. Only the three fully connected layers above it, ending in the 8-way output, see both patches at once. That forces most of the reasoning to happen inside each patch's own representation, which is the part we keep.

The interesting part is what they had to forbid. If neighboring patches touch, the network can match edges and textures across the border and never look at the content. So they left a gap of 48 pixels, half a patch, between cells. Long straight lines could still give the answer away across the gap. So they also shifted each patch at random by up to 7 pixels in each direction, a random shift called jitter. The exercise builds exactly this sampler.

import numpy as np

def relative_position_pair(img, P, gap, jitter, rng):
    raise NotImplementedError

Jigsaw puzzles

A jigsaw task shuffles the pieces of a photo and asks which shuffle was used. Noroozi and Favaro (2016) used it to ask about many spatial relations at once. They crop a 225 by 225 region and split it into a 3 by 3 grid of 75-pixel cells. From a random spot in each cell they take a 64 by 64 tile, which leaves random gaps between tiles. They shuffle the 9 tiles with one permutation from a fixed set, and the network must say which permutation it was.

Each tile goes through the same convolutional tower, and only the last fully connected layers see all nine. Because the tiles are seen together, one ambiguous tile can be placed by elimination. The authors argued that this teaches the network what parts an object is made of.

Which permutations go in the set matters. Two permutations that differ only by swapping two similar tiles, say two patches of sky, are impossible to tell apart, so that label would be noise. So the authors chose permutations that are far apart in Hamming distance, the number of positions at which two permutations differ. They scored the features by fine-tuning a detector on PASCAL VOC 2007, a 20-class object detection benchmark. The score is mean average precision (mAP), the standard detection score from 0 to 100.

Their Table 4 shows the effect of the distance. With 1,000 permutations, an average Hamming distance of 8.00 gave 71 percent puzzle accuracy and 53.2 mAP. Sets with averages of 6.35 and 3.99 gave 62 and 54 percent on the puzzle and only 51.3 and 50.2 mAP. One fact puts 8.00 in perspective.

Proof. At each position $i$, the item $\pi'(i)$ is uniform over the $n$ items and independent of $\pi(i)$, so the two agree there with probability $1/n$. By linearity of expectation they agree at $n \cdot (1/n) = 1$ position on average, so they differ at $n - 1$. ∎

For 9 tiles that is exactly 8, so an average of 8.00 is what a random set already gives. The paper's rule builds the set greedily. It adds, one at a time, the permutation with the largest summed distance to the permutations already chosen. That pushes the average barely above 8, and the exercise shows that it leaves the closest pair as close as two permutations can be.

import numpy as np
import itertools

def permutation_set(n_tiles, N, rng, rule="sum"):
    raise NotImplementedError

Run your solution with 9 tiles and $N = 100$ and it reproduces a row of the paper's Table 4, an average distance of 8.08 and a minimum of 2, which is a single swap. The farthest-point rule raises the minimum to 6 at almost the same average, about 8.02. A 9-tile run scans all 362,880 permutations 100 times, so it takes a few seconds in plain Python and longer in the browser.

Which mattered more for the features? In the same table, smaller sets with larger minimum distances (3 to 6) gave about the same detection mAP, 52.4 to 52.8, while lowering the average hurt. The tiny sets with the largest minima, 6 to 10 permutations at distance 7 to 9, made the puzzle too easy (97 to 99 percent) and fell to about 49.5 mAP. So for jigsaws the average mattered more than the minimum, and the set must not be so small that the task becomes trivial. When you need a set of well-separated codes in your own work, though, farthest-point sampling is the rule to reach for.

Rotations

Rotation prediction, the example we started from, was proposed by Gidaris, Singh and Komodakis (2018) as the simplest task of all. It turns a whole photo by a multiple of 90 degrees and asks which multiple. To know which way is up, the network must find skies, faces, text, and the bottoms of objects.

Multiples of 90 degrees have a practical advantage. They are flips and transposes of the pixel grid, so no interpolation is needed and no resampling artifacts give the angle away. The authors also found it better to put all four rotated copies of each image in the same batch than to sample one rotation per image. Their AlexNet trained in about two days on one GPU.

The cell builds such a batch and then asks a question that will come back several times. How much of the task can be solved from something other than the objects? In a photo whose sky is brighter than its ground, the mean brightness of the four borders already gives the rotation away.

Four numbers solve rotation perfectly when the scene has a bright top, and fall to chance on textures with no up. Real photos sit in between. That is why rotation prediction works at all, and also why it learns less than we would like. Whatever cheap cue solves a pretext task, the network will find it first.

Colorization and inpainting

Two tasks predict pixels rather than a small label. Zhang, Isola and Efros (2016) gave the network only the lightness of each pixel and asked for its color. They worked in the Lab color space, where $L$ is lightness and $a$ and $b$ are two color coordinates, so the input is $L$ and the target is $(a, b)$. A gray shirt could be red or blue, and averaging the possible colors would give a dull brown. So they treated color as classification. They cut the $(a, b)$ plane into squares of side 10 and kept the 313 squares that real colors occupy. They also reweighted the loss toward rare colors, so that the network would not settle for a safe brownish gray. Colorization is a cross-channel prediction, one part of each pixel predicting another.

Inpainting removes a region of the image and asks the network to paint it back. Pathak et al. (2016) trained a context encoder to fill the hole from its surroundings. Its loss was a pixel-wise squared error plus, for sharper results, an adversarial loss, the score of a second network trained to tell real fillings from generated ones. Predicting the contents of a hole requires knowing what objects look like. The idea returned five years later, with patches and a much larger fraction removed, as the masked autoencoders of §28.5.

Shortcuts, and why pretext accuracy is not the goal

A pretext task is a stand-in for what we want, and the network optimizes the stand-in. A shortcut is any regularity that predicts the target without understanding the image. The network will learn it, because it is the easiest way down the loss. The most famous example came from the very first context-prediction experiments.

Doersch et al. noticed that for some images the network had found another trivial solution. When they searched feature space for a patch's nearest neighbors, some patches retrieved patches from the same absolute position in other photos, whatever their content. The cause was chromatic aberration. A lens focuses different wavelengths slightly differently. In some cameras one color channel, commonly green, is shrunk slightly toward the image center relative to the others. The offset between green and magenta (red plus blue) then grows with the distance from the center, so a patch's color fringes reveal where it sat in the frame. Once the network knows both patches' absolute positions, their relative position is arithmetic.

The authors removed the cue in two ways. One projects every pixel's color off the green-magenta axis $\mathbf{a} = (-1, 2, -1)$, a row vector, by multiplying it by $\mathbf{I} - \mathbf{a}^\top\mathbf{a}/(\mathbf{a}\mathbf{a}^\top)$. The other drops two of the three color channels at random and replaces them with faint noise.

The shift inside a patch can be estimated without any learning. Suppose green at each pixel shows what red shows a small displacement $\mathbf{d}$ away, $G(\mathbf{u}) = R(\mathbf{u} + \mathbf{d})$. A first-order Taylor expansion gives $G(\mathbf{u}) \approx R(\mathbf{u}) + \nabla R(\mathbf{u})^\top\mathbf{d}$ at every pixel $\mathbf{u}$, which is linear in $\mathbf{d}$. So the best $\mathbf{d}$ in the least-squares sense over the patch solves a 2 by 2 linear system.

A fringe of 1 percent solves the task perfectly from two numbers. Here that fringe is a difference of only about half a pixel between the shifts of two side-by-side patches, since their centers are 48 pixels apart and $0.01\times 48 = 0.48$. The texture has no objects, so all of that accuracy came from the lens. The projection removes the cue completely and brings the accuracy back to chance.

A second, subtler problem is that pretext accuracy does not rank networks. Kolesnikov, Zhai and Beyer (2019) ran a large controlled study of four pretext tasks on three families of networks at several widths. The families were ResNets, which have skip connections (§0.8), RevNets, a reversible variant of ResNets, and VGG, a plain stack of convolutions with no skip connections. Pretext accuracy was somewhat predictive of downstream quality within one family, but not across families. The widest VGG was the best rotation predictor and gave poor features.

They found two more patterns. In networks without skip connections, such as VGG and the AlexNets of the original papers, the representation got worse in the last layers, which specialize to the pretext. In ResNets it did not. And widening the network, which also widens the representation, helped every method. So architecture mattered as much as the task. Context prediction, one of the earliest methods, reached 51.4 percent on an ImageNet linear probe and beat every self-supervised result published before the study once it ran on the right network.

In practice, never pick a pretext model by its pretext accuracy. An interviewer who asks "is pretext accuracy a good measure of quality?" wants the jigsaw ablation and the VGG result as evidence for "no".

Invariance and equivariance

The pretext tasks so far share a structure. They ask the representation to change in a predictable way when the input is transformed, since a network can report the rotation only if the rotation shows up in its features. The opposite request, that the representation not change, is what the rest of the chapter uses. Both need the idea of a random transformation of an image.

In plain words, an invariant feature ignores the transformation. An equivariant one changes with it in a fixed way that does not depend on the image. Rotation prediction needs features that change with rotation, ideally equivariantly, so that a head can read the angle off them. Contrastive learning asks for features that do not change under crops and color jitter.

Misra and van der Maaten (2020) compared the two directly. Their method, PIRL (Pretext-Invariant Representation Learning), used the jigsaw shuffle not as a target to predict but as an augmentation to be invariant to. It pulled the representation of an image toward that of its shuffled version. With a ResNet-50 on ImageNet, the invariant version reached 63.6 percent with a linear probe, against 45.7 for the same network trained to predict the shuffle.

Every augmentation is a claim about what does not matter. Random crops claim that a part identifies the whole. Color jitter claims that hue is irrelevant. Flips claim that left and right are interchangeable. When a claim is wrong for a downstream task, the representation has been trained to throw away exactly what that task needs.

This has been measured too. Ericsson et al. (2021) evaluated 13 self-supervised models on 40 downstream tasks. When they reconstructed images from the features, the self-supervised ones gave back less faithful color than a supervised network, and the authors attribute this to the heavy color distortion in training. That matters for flowers, birds, or medical stains. §28.2 builds the concrete augmentation recipe and measures each part. In practice, list what your downstream task needs before you pick augmentations, and remove any augmentation that destroys it.

Every image is its own class

The idea that turned self-supervised vision from a collection of puzzles into a field was to take classification to its extreme. Dosovitskiy et al. (2014) made each randomly sampled image patch a surrogate class. They generated many transformed copies of each patch and trained a classifier to tell the classes apart. The features then had to survive the transformations and still distinguish the patches.

With thousands of classes this was feasible. With a million it was not. A softmax layer over a million classes has one weight vector per class. Each class is seen only once per epoch, so those weights barely train.

Wu et al. (2018) removed the class weight vectors. If every image is a class, let the image's own stored feature serve as that class's prototype.

In plain words, each image must be recognizable among all the others from its own feature. The features of the other images, read from a slowly updated table, play the competing classes. "Non-parametric" means there are no learned class weights, only stored features. The temperature $\tau$ divides the scores before the softmax, so a small $\tau$ makes the weights sharper (§7.2 analyzes it in detail).

Wu et al. used $d = 128$ and $\tau = 0.07$. With unit vectors the scores then lie in $[-1/\tau, 1/\tau] = [-14.3, 14.3]$, and a small $\tau$ is what lets the softmax pick out one image among a million. They did not compute the full sum over all $n$ images. They used noise-contrastive estimation (Gutmann and Hyvärinen 2010), which turns the softmax into a yes-or-no task, telling the true entry apart from 4,096 entries drawn at random from the bank.

Why would telling individual photographs apart teach anything about categories? The cheapest way to tell 1.28 million images apart under random crops and color changes is to describe what is in them. Two photos of tabby cats then end up near each other because they share most of their description, not because anyone said "cat".

Wu et al. measured this. On CIFAR-10 their non-parametric features gave 80.8 percent accuracy with a nearest-neighbor classifier, against 63.0 percent for the same network trained with an ordinary softmax that has one weight vector per instance. On ImageNet a ResNet-50 reached 54.0 percent with a linear probe on its frozen features. The memory bank is the engineering trick that makes a million classes affordable.

The bank needs a rule for how entries change. Wu et al. added a proximal term, a penalty on each feature for moving far from its stored entry. The more common form, which MoCo's authors describe as a momentum update on the bank, blends the old entry with the new feature and renormalizes,

with a momentum $m \in [0, 1)$. In plain words, the entry moves a fraction $1 - m$ of the way toward the new feature and is then put back on the unit sphere. The exercise uses this form, with the exact softmax over a small bank.

import numpy as np

def instdisc_step(h, idx, bank, tau, m):
    raise NotImplementedError

Instance discrimination is a contrastive loss in disguise. Recall from §7.2 that the InfoNCE loss scores one positive, the item that truly matches, against many negatives. It applies a softmax cross-entropy with the positive as the correct class. Here the positive for a view of image $i$ is image $i$'s bank entry, and the negatives are everyone else's. The step from here to the contrastive methods of §28.2, SimCLR and MoCo, is small. Replace the stored entry by the feature of a second, freshly augmented view of the same image.

An interviewer may ask how you would train a classifier with a million classes and one example each. The answer is this section's trick. Drop the class weights, store one feature per class, and approximate the softmax with sampled negatives.

Augmentations say what should not matter

Take two views of one photograph, and a loss that wants their representations to agree. This is the template behind nearly every method of the next three sections, and it has one trap built in. If agreement is all the loss asks for, an encoder that maps every image to the same vector agrees perfectly. That failure is called collapse. §1.10 watched it happen with a loss that only pulled pairs together, and §7.2 showed that pushing other examples away prevents it. The methods differ mainly in what they use instead of, or in addition to, that push.

The table lists how each family of methods rules out the constant solution, and where the book builds it.

Notice what is absent from the template. Nothing says which images belong together except the augmentations. If two photos of different dogs end up close, no label put them there. Something in the augmentations, and in the images themselves, did. The rest of this section makes that sentence precise, and it is the theoretical reason the whole approach works.

The augmentation graph

HaoChen, Wei, Gaidon and Ma (2021) found the right object to think about. Picture every possible view of every possible image as a point, a vertex of a graph. Join two views by an edge whose weight is the probability that they come out of the same image.

Crops of one cat photo are joined directly. Two different cat photos are joined indirectly. After cropping, blur, and color jitter, a tight crop of one cat could have come from either of two similar cat photos, so the two photos share views. A chain of such shared views links all the cat photos. A cat and a truck, if the augmentations never make them look alike, sit in different pieces of the graph.

In this part of the section we follow their notation. We write $\bar x$ for a natural image, $x$ for a view, and $\mathcal{A}(x\mid\bar x)$ for the probability that the augmentation turns $\bar x$ into $x$. We take the set of views to be finite with $N$ elements. That is true of real images, which have finitely many pixel values, and it keeps every step elementary.

In plain words, $\Omega_{xx'}$ is how often the pair $(x, x')$ shows up when you take one random photo and augment it twice. Dividing by the square roots of the degrees stops very common views from dominating.

Two properties come straight from the definition. $\boldsymbol\Omega$ is symmetric. It is also positive semidefinite, meaning $\mathbf{u}^\top\boldsymbol\Omega\mathbf{u} \ge 0$ for every vector $\mathbf{u}$, because $\mathbf{u}^\top\boldsymbol\Omega\mathbf{u} = \mathbb{E}_{\bar x}\big[\big(\sum_x \mathcal{A}(x\mid\bar x)\,u_x\big)^2\big]$ is an average of squares. With finitely many images, write $\mathbf{p}$ for their probabilities and $\mathbf{T}$ for the image-to-view matrix with entries $T_{\bar x x} = \mathcal{A}(x\mid\bar x)$. Then $\boldsymbol\Omega = \mathbf{T}^\top\operatorname{diag}(\mathbf{p})\,\mathbf{T}$, a weighted Gram matrix, a matrix of inner products in the sense of §0.10. So $\bar{\boldsymbol\Omega}$ is symmetric positive semidefinite too. It has real eigenvalues $\gamma_1 \ge \gamma_2 \ge \dots \ge \gamma_N \ge 0$ and orthonormal eigenvectors $\mathbf{u}_1, \dots, \mathbf{u}_N$.

The cell builds this graph from $\mathbf{T}$ and $\mathbf{p}$ and checks the spectrum for several $q$.

The top eigenvectors found the classes without any label. That is the pattern the rest of the section turns into a theorem.

The spectral contrastive loss

Spectral clustering groups the vertices of a graph using the top eigenvectors of its normalized adjacency matrix. Those eigenvectors, divided by the square roots of the degrees, are nearly constant on groups that are well connected inside and weakly connected to each other. That is what happened in the example. But we cannot write the augmentation graph down, since it has a vertex for every possible view. HaoChen et al. observed that a simple loss on pairs of views, which needs only samples, has exactly these eigenvectors as its minimizers.

Let $f$ map each view to $\R^k$. It stands for the whole network that produces the embedding, encoder and any head together. A positive pair $(x, x^+)$ is two independent views of one image $\bar x \sim \mathcal{D}$. A negative pair $(x, x^-)$ is two independent views of two independently drawn images. So $x$ and $x^-$ are independent, and each has the marginal distribution $\boldsymbol\omega$.

Two matrix facts are needed. The Frobenius norm $\lVert\mathbf{A}\rVert_F$ is the square root of the sum of squared entries, and $\langle\mathbf{A}, \mathbf{B}\rangle = \sum_{ij}A_{ij}B_{ij}$ is the matching inner product. The Eckart-Young theorem of §0.10 says that the best rank-$k$ approximation of a matrix in this norm keeps its top $k$ singular directions. The squared error is the sum of the squared singular values left out.

The proof has two steps. The positive term becomes an inner product with $\bar{\boldsymbol\Omega}$, and the negative term becomes the squared norm of $\mathbf{F}\mathbf{F}^\top$. Completing the square then gives the factorization, and Eckart-Young gives the bound.

In plain words, minimizing this loss finds the best rank-$k$ approximation of the augmentation graph, which is what spectral clustering computes. The first term rewards agreement between views of one image. The second penalizes agreement between unrelated views, quadratically, and that is what rules out the constant solution. A constant $f = \mathbf{c}$ pays $-2\lVert\mathbf{c}\rVert^2 + \lVert\mathbf{c}\rVert^4$, which is never below $-1$. The optimum is $-\sum_{i\le k}\gamma_i^2$, close to $-k$ when the graph splits into $k$ pieces. In the hand example at $q = 0.1$ with $k = 2$, collapse scores $-1$ and the class indicators score $-2$.

The loss never mentions eigenvectors, and it can be estimated from samples. The first term uses two views of each image in a batch, and the second uses views of different images. That is what makes it trainable.

When a linear probe is perfect

Why is the top of the spectrum the right thing to learn? Because of what an eigenvalue of 1 means. A connected component of the graph is a piece with no edge of positive weight leaving it.

The proof of items 1 and 2 rests on one identity. For any vector, the matrix $\mathbf{I} - \bar{\boldsymbol\Omega}$, called the normalized graph Laplacian, gives a weighted sum of squared differences across edges. That sum is never negative. It is zero exactly when the vector, after rescaling by the degrees, is constant on each component.

Item 3 needs more care, because the eigenvalues just below 1 may tie, and then the minimizer is not unique. The derivation shows that every minimizer still keeps the whole eigenspace of eigenvalue 1.

In plain words, augmentations draw a graph, and the graph's connected pieces show up as eigenvectors with eigenvalue 1. Given at least one dimension per piece, the spectral contrastive loss puts each piece at its own point of $\R^k$, where a linear classifier can read it off. Which images share a piece is decided by the augmentations and the data, never by labels. So the augmentations are the supervision.

The theorem also says what goes wrong. If augmentations are too weak, each image is its own piece. Then there are far more eigenvalue-1 directions than dimensions, and the loss keeps an arbitrary few of them. If augmentations leak across classes, the class pieces merge and the class eigenvalues drop below 1. Once they fall below the directions that split a class internally, the loss prefers the wrong ones.

The cell tests both failures. It builds an augmentation graph for 300 points in three hidden classes. The augmentation adds Gaussian noise of standard deviation $\sigma$, the running example of HaoChen et al. A leak sends a fraction $q$ of each image's views to views of other classes.

The probe is fit and scored on the same 600 views, so it measures whether the classes are linearly separable in $f$, not whether the probe generalizes. With $\sigma = 0.05$ the noise is too small to connect different images. The top eigenvalues are all 1 to four decimals, the three kept directions are arbitrary, and the probe gets 66 percent. With $\sigma = 0.4$ each class is nearly one piece, joined to the others only by tiny Gaussian tails. $\gamma_2$ and $\gamma_3$ stay above 0.996 while $\gamma_4$, the first direction inside a class, is 0.68, and the probe reaches 98.7 percent.

As $q$ grows, the class eigenvalues fall toward the within-class ones, 0.73 against 0.49 at $q = 0.1$ and 0.35 against 0.27 at $q = 0.3$. When the gap closes at $q = 0.45$, the probe drops to 47 percent. The widget lets you move through these regimes yourself.

What this changes in practice is how you tune augmentations. They must be strong enough to connect the views of each class and weak enough never to connect different classes. An interviewer may ask "why do contrastive methods need such strong augmentation?", and this picture is the answer.

The eigenvectors above are a nonparametric answer, one free vector per view, which we could compute only because the graph was tiny. A network gives a parametric $f$, a function with weights, and learns from sampled pairs. The next exercise writes the loss it learns from.

import numpy as np

def spectral_loss(Z1, Z2):
    raise NotImplementedError

Now train a real parametric $f$ with this loss and see whether it finds what the theorem predicts. The cell uses four blobs of points at the corners of a square, labeled like XOR, with one class on each diagonal. Each class is then a union of two components, and no linear probe on the raw coordinates can separate them.

The encoder must be nonlinear, but we want to train it with plain gradient steps on one weight matrix. So it first maps each input through 128 fixed random Fourier features, $\phi(\mathbf{x}) = \sqrt{2/D}\,\cos(\mathbf{x}\mathbf{W}_{\text{rff}} + \mathbf{b}_{\text{rff}})$ with $D = 128$ and a random $\mathbf{W}_{\text{rff}}$ and $\mathbf{b}_{\text{rff}}$ drawn once (Rahimi and Recht 2007). Then $f(\mathbf{x}) = \mathbf{W}^\top\phi(\mathbf{x})$ is a nonlinear function of the input, and only $\mathbf{W}$ is trained. It never sees a label.

Everything the theorem says is visible. The final batch loss estimates $\mathcal{L}(f)$, which can never fall below $-\sum_{i\le k}\gamma_i^2 \ge -k$. Training reaches about $-1.9$ for $k = 2$ and $-3.8$ to $-3.9$ for $k = 4$, so the graph's top four eigenvalues are all close to 1, one per blob. With $k = 4$, at least the number of blobs $r = 4$, the probe separates the XOR classes at 99 percent for every seed, while the raw coordinates give 50 percent.

With $k = 2 < r$ the loss cannot tell which two of the four blob directions to keep, since they are all worth the same. So the result depends on the random start, 99, 52, and 95 percent for three seeds. Gradient descent on a sampled loss found the spectral answer without ever forming the graph. In practice this is why embedding width matters, and why §28.3 finds that wider projection heads help.

import numpy as np

def augmentation_graph(T, p):
    raise NotImplementedError

def spectral_features(Omega, k):
    raise NotImplementedError

What the theorem assumes

The theorem is exact, but each of its assumptions is violated in practice, so it is worth listing them.

• It describes the population graph, every possible view of every possible image. The graph of the training set alone would have almost no edges, since two training images essentially never produce the identical crop.
• It assumes $f$ reaches the global minimum. A real encoder is a finite network trained for a finite time. Its built-in preferences, called its inductive biases, decide how the loss extends to views it never saw.
• It assumes class pieces are exactly disconnected, and real ones never are. HaoChen et al. also prove a bound for the nearly disconnected case (their Theorem 3.8), with at least twice as many dimensions as classes. There the probe error is controlled by how little edge weight crosses classes compared with how well each class holds together. They also show that the loss can be minimized from finite samples with guarantees, but the constants are not ones you would quote in an interview.

What carries over is the qualitative picture, which the cells and the widget show. Augmentations must connect each class and must not connect different classes.

How a representation is graded

Every number in the rest of the chapter comes from one of three protocols. Each takes a pretrained encoder and a labeled downstream dataset. §28.6 treats them in depth, with their recipes, their noise, and many variants. Here are the definitions you need to read a results table.

The protocols measure different things. A linear probe asks whether the classes can be read off the features by a linear function. k-NN asks whether neighbors share a class, which is a question about the local geometry. It needs no training at all, which makes it the cheap check people run during pretraining. Fine-tuning asks how good a starting point the weights are. It can reward representations that are neither linear nor locally clustered, as masked autoencoders will show in §28.5. The cell compares the first two on the XOR data.

The raw coordinates fail the linear probe (50 percent) and pass k-NN (99 percent), because XOR classes are locally coherent but not linearly separable. The learned features pass both. When two protocols disagree about a representation, they are telling you different true things about it. An interviewer who asks which protocol to report wants exactly this distinction, and usually wants both numbers.

Since 2018 the linear probe on ImageNet has been the headline number, and its trend is the story of this chapter.

In PyTorch

The torch block does three things. It checks autograd against the closed-form gradient of Exercise 28.1d. It trains a small MLP encoder with the spectral contrastive loss on the XOR blobs, with Gaussian noise as the augmentation. And it shows a memory-bank update done the way real code does it, under torch.no_grad(), so the bank never enters the computation graph.

Choosing a source of supervision

The families below cover the whole chapter and its neighbors. Each is defined by what it predicts, and each has a characteristic way to fail.

A closely related idea keeps the "predict a hidden part" of masked modeling but predicts the embedding of the hidden region instead of its pixels. That is the idea behind joint-embedding predictive architectures, a family that combines the dense targets of the second row with the semantic features of the fourth.

Read more

• Doersch, Gupta & Efros (2015). Unsupervised Visual Representation Learning by Context Prediction. The relative-position task, and Section 3.1 on the gap, the jitter, and the chromatic aberration shortcut.
• Noroozi & Favaro (2016). Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. Algorithm 1 for the permutation set, and Tables 4 and 5, the clearest measurement of how shortcuts trade pretext accuracy for feature quality.
• Gidaris, Singh & Komodakis (2018). Unsupervised Representation Learning by Predicting Image Rotations. The simplest pretext task and the argument for multiples of 90 degrees.
• Wu, Xiong, Yu & Lin (2018). Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination. Every image its own class, the memory bank, and the weighted k-NN evaluation still in use.
• Kolesnikov, Zhai & Beyer (2019). Revisiting Self-Supervised Visual Representation Learning. A controlled study showing that architecture and width matter as much as the pretext task, and that pretext accuracy is a poor guide.
• HaoChen, Wei, Gaidon & Ma (2021). Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. The augmentation graph, the spectral contrastive loss, and the probe guarantees for nearly disconnected graphs.
• Goyal et al. (2021). Self-supervised Pretraining of Visual Features in the Wild. SEER, the first demonstration that random uncurated images at billion scale are enough.
• Balestriero et al. (2023). A Cookbook of Self-Supervised Learning. The best single survey of the methods, losses, and practical settings of this chapter.

Watch next

▶

▶

▶