Masked image modeling trains a vision network without labels by hiding most of a photograph and asking the network to fill in what is missing. The photo is cut into small squares called patches. A random three quarters of them are hidden, the network sees the rest, and it is graded only on its guesses about the hidden ones. To guess well it has to learn what the things in pictures look like, and that knowledge is what we keep and reuse.

The idea comes from language. BERT, the text encoder of §7.1, hides 15 percent of the words of a sentence and predicts each hidden word from the words around it. Moved to images, that recipe became the masked autoencoder, or MAE (He et al. 2022). It pretrains large Vision Transformers at about a third of the compute that the same model would need if it read the hidden patches too.

MAE pretrained the image encoder of Segment Anything, the model that cuts out any object you click on (Kirillov et al. 2023). The same recipe trains video models (§29.4) and speech models (§35.5). And DINOv3, among the strongest self-supervised image encoders published, still adds a masked-patch loss to its objective. Interviewers ask about MAE often, because each of its design choices has a reason you can derive at a whiteboard.

The earlier methods of this chapter work differently. Contrastive learning and self-distillation take two distorted copies of one photo and train the network to give them matching summary vectors, one vector per image. Masked modeling distorts nothing. It grades every hidden patch separately, so one image gives well over a hundred training signals.

One measurement shows what is different about images. MAE's authors pretrained a large Vision Transformer, ViT-L, several times, each time with a different fraction of its patches hidden. They then tested each network on ImageNet, a benchmark of 1.28 million labeled photos in 1,000 classes, in two ways. A linear probe freezes the network and trains only a linear classifier on its output features (§28.1). Fine-tuning trains the whole network further on the labels.

The linear probe climbs by almost 20 points from 10 to 75 percent and falls after that. A ratio near BERT's 15 percent sits at the bottom of the curve. Why does an image need three quarters of itself hidden, when a sentence needs only about a seventh?

This section answers that question with a probability you can compute by hand and a simulation you can run. Then it builds MAE piece by piece and counts exactly how much compute it saves by never showing its encoder a hidden patch. Then it surveys what else a masked model can predict, from pixels to the codes of an image tokenizer to the features of a teacher network. It also covers models that predict patches one after another, like a language model. It ends with a puzzle visible in the table above. MAE's features do much worse under a linear probe than under fine-tuning, a disagreement that §28.6 takes apart.

From masked words to masked patches

Two ideas from earlier chapters meet here. The first is how a Vision Transformer reads an image (§27.1). It reads it the way a language model reads text, as a sequence of tokens. Take a $224\times224$ color photo and cut it into squares of $16\times16$ pixels. That gives a $14\times14$ grid of 196 patches, and each patch holds $16\cdot16\cdot3 = 768$ numbers, the red, green, and blue values of its pixels. The patches are read in raster order, left to right along each row and then row by row, like text on a page. A linear map turns each patch into a vector of width $d$, called a token, and a Transformer, a stack of attention blocks, reads the tokens.

In symbols, an image $x$ with $C$ channels, height $H$, and width $W$ is cut into $N_p = HW/P^2$ square patches of side $P$ pixels. Patch $i$ is flattened into a vector $\mathbf{x}_i \in \R^{P^2C}$, with the $C$ channel values of each pixel kept together.

The second idea is masked language modeling (§7.1). Pick a set of positions, hide what is there, predict it, and take the loss only at those positions. Here are the image versions, made precise.

A masked image model then does three things. It corrupts the image according to $\mathcal{M}$, runs an encoder $f_\theta$ (a Vision Transformer with parameters $\theta$), and predicts, for every hidden patch $i \in \mathcal{M}$, some target $\boldsymbol\phi_i(x)$ computed from the uncorrupted image.

In plain words, hide some patches, make the network guess something about each hidden one, and grade only the guesses about what was hidden.

This objective differs from the rest of the chapter in two ways that matter.

It keeps different information. Contrastive learning (§28.2) trains two randomly distorted views of one photo, for example one recolored and one cropped, to give the same vector. So its features learn to ignore whatever the distortions change, such as color and framing. Masked modeling distorts nothing. Its features must keep whatever helps predict a hidden patch from the visible ones, color included.

With pixel targets it cannot collapse. Collapse (§28.3) is the failure in which a network outputs the same vector for every image. A loss that only compares the network with itself can still be perfect then. Here the target is fixed by the image, so a constant prediction scores badly. Collapse comes back only when the target is made by a network that is also being trained. That is why the methods late in this section that predict a teacher network's features reuse the safeguards of DINO (§28.4).

Why an image needs most of its patches hidden

Text is dense. Hide one word of a sentence and the remaining words constrain it but rarely pin it down, so hiding 15 percent already makes a hard task. Images are redundant. Neighboring pixels have nearly the same color, because surfaces are smooth and lighting changes slowly. A hidden patch whose neighbor is visible can be filled by continuing the neighbor's colors and edges, and that needs no idea of what the image shows. A hidden patch whose neighbors are all hidden cannot be filled that way.

A standard measurement backs this up. The power spectrum of an image says how much of its variation sits at each spatial frequency $k$, from slow waves across the whole image to ripples one pixel wide. For natural photographs it falls roughly like $1/k^2$ (Field 1987). At that rate each band of frequencies from $k$ to $2k$ holds about the same amount of variation, whatever $k$ is. On a $224\times224$ image that follows this law, about 58 percent of the variation is in waves longer than a 16-pixel patch, and neighboring patches share those waves.

So the question becomes a count. Call a hidden patch isolated when its four grid neighbors (up, down, left, and right) are hidden too. How often is a hidden patch isolated? Here is the count on MAE's grid, before the general formula.

The same count works on any grid and at any ratio. It also has a simple approximation worth memorizing, the masking ratio raised to the fourth power.

The cell checks the formula against a simulation of MAE's own masking rule on the $14\times14$ grid.

The exact value sits slightly below $\rho^4$ at moderate ratios, because once $i$ is hidden, fewer hidden patches remain for its neighbors. At $\rho = 0.9$ it sits slightly above, because the floor in $T_v$ hides a little more than $0.9N_p$.

What this changes in practice is the choice of ratio. At 15 percent, a network can lower its loss almost everywhere by blending its neighbors, which teaches it nothing about content. Only a high ratio makes a sizable share of the targets unreachable that way. An interviewer who asks "why 75 percent and not 15" wants this argument, ideally with the $\rho^4$ estimate.

SimMIM (Xie et al. 2022), a stripped-down masked image model that appears later in this section, measured the same idea pixel by pixel.

SimMIM swept masking ratios and square sizes and plotted the accuracy after fine-tuning against AvgDist. The best runs had an AvgDist between 10 and 20 pixels, and accuracy was worse on both sides. When the distance is too small, the network can learn to copy nearby pixels. When it is too large, too little is left to reason from. SimMIM chose 32-pixel squares at a ratio of 0.6 as its default.

Now let us watch the copying shortcut fail. The next cells build smooth random images and fill every hidden patch with a copy of the nearest visible patch. That predictor knows nothing about content, so it shows how far copying alone can go.

The images are Gaussian random fields. Each is random noise reshaped in the Fourier domain so that coarse variations are strong and fine ones are weak. Here the power falls like $1/k^3$, so the images are smoother than a photograph, with about 96 percent of their variation in waves longer than a patch. They are $224\times224$ with 16-pixel patches. Quality is measured by $R^2 = 1 - \text{SSE}/\text{SST}$. SSE is the squared error of the copy, and SST is the squared spread of the true hidden pixels around their mean. So $R^2 = 1$ is a perfect copy, and $R^2 = 0$ is no better than predicting the mean.

On real photographs the same logic holds, with one difference. A random field has no objects, so nothing beyond local smoothness can help. A photograph does have objects. The hidden third of a dog's head is predictable from the visible two thirds, but only to a model that knows what dogs look like. That knowledge is the representation we are after.

Move the ratio and watch the outlined patches, the hidden ones with no visible patch above, below, or beside them. Compare their copies with the image on the left, and their share with $\rho^4$. Then switch to grid masking. A third of its hidden patches are outlined, yet each of them touches a visible patch at a corner, so the copies stay close. Blockwise masking, the last option, isolates patches even at low ratios. The exercise below measures the same effect pixel by pixel, with SimMIM's AvgDist.

import numpy as np

def nearest_visible(img, visible):
    """img (H, W), visible (H, W) bool. Return (prediction (H, W), distance to nearest visible pixel (H, W))."""
    raise NotImplementedError

def masking_sweep(imgs, P, rhos, seed):
    """Return {"r2": array over rhos, "avgdist": array over rhos}."""

The masked autoencoder

An autoencoder is the simplest network that learns without labels. An encoder maps the input to a latent, a compressed internal representation. A decoder maps the latent back to the input, and training minimizes the reconstruction error. On its own an autoencoder can learn to pass its input straight through, which teaches nothing. Useful autoencoders make the task harder. One way squeezes the latent through a narrow bottleneck. Another corrupts the input and asks for the clean version, which is a denoising autoencoder (Vincent et al. 2008). Autoencoders in general get their own chapter. A masked autoencoder is a denoising autoencoder whose corruption deletes most of the input.

The first image version of this idea is older than Vision Transformers. A context encoder (Pathak et al. 2016), one of the pretext tasks of §28.1, cut one large hole in the middle of a photograph and trained a convolutional network to paint it back. It worked, but only a little. The masked autoencoder changes three things. It hides many patches instead of one hole, it removes far more of the image, and its encoder never sees what was removed.

Two design decisions by He et al. (2022) turned this old idea into a method that scales. First, the encoder sees only the visible patches, with no placeholder for the hidden ones, so its cost falls as the ratio rises. Second, the decoder, which does see placeholders, is small and is thrown away after pretraining. Because the encoder and the decoder see different inputs and have very different sizes, the paper calls the design asymmetric.

In plain words, the encoder looks at a quarter of the image, the decoder is told where the other three quarters were, and the pair is graded on painting them back.

Random masking by sorting noise

How do you pick a random subset of exactly $T_v$ patches for every image in a batch, on an accelerator, without a loop? MAE draws one uniform random number per patch and sorts. The patches with the $T_v$ smallest numbers are kept.

Here is a tiny case. With $N = 4$ patches and noise $u = (0.7, 0.2, 0.9, 0.4)$, sorting gives the order $\pi = (1, 3, 0, 2)$. Patch 1 has the smallest number, then patch 3, and so on. Keeping $T_v = 2$ keeps patches 1 and 3. To undo the shuffle later we need, for each patch, its position in that order. Patch 0 is at position 2, patch 1 at 0, patch 2 at 3, and patch 3 at 1, so the restore index is $(2, 0, 3, 1)$. Sorting $\pi$ itself gives exactly that.

The cell confirms both claims. With $N = 6$ patches and two kept there are $\binom{6}{2} = 15$ possible visible sets, and each should appear about one fifteenth of the time.

All fifteen frequencies fall between 0.064 and 0.069, against 0.0667, and the restore index inverts the shuffle in every trial. The batched version is a standard interview question. It gathers along the token axis and returns the binary mask in the original order, and it is the next exercise.

import numpy as np

def random_masking(X, rho, rng):
    """X (B, N, d) -> visible tokens (B, n_keep, d), mask (B, N) with 1 = masked, ids_restore (B, N)."""
    raise NotImplementedError

The loss, with each patch normalized by itself

MAE's decoder predicts pixel values, and the loss is the mean squared error on the hidden patches. Its best version changes the target first. Each target patch is standardized by its own mean and spread, so that its values average 0 and have spread 1.

Here is a tiny case. A patch with the four values $(2, 4, 6, 8)$ has mean 5. Its unbiased variance is $\big((-3)^2 + (-1)^2 + 1^2 + 3^2\big)/3 = 20/3$, so its normalized form is $(-3, -1, 1, 3)/\sqrt{20/3} \approx (-1.16, -0.39, 0.39, 1.16)$. Doubling every value gives the patch $(4, 8, 12, 16)$, which is brighter and has more contrast, and it has exactly the same normalized form. Brightness and contrast are gone, and only the pattern is left.

In general, for a patch $\mathbf{x}_i$ with $n = P^2C$ values $x_{i1}, \dots, x_{in}$, let

with $\epsilon = 10^{-6}$ to avoid dividing by zero on a flat patch. The mean and the variance run over all $n$ values of the patch, channels together. The variance is the unbiased one, dividing by $n - 1$, because MAE's code calls PyTorch's var, whose default does that.

In plain words, for each hidden patch, subtract its average brightness, divide by its contrast, and ask for the pattern that remains, pixel by pixel.

Why does removing a patch's own brightness and contrast help? Those two numbers are the most predictable part of a hidden patch, since smooth images give them away through the neighbors. They are also the part that says least about content. Normalizing takes them out of the target, so the loss is spent on the edges and texture inside the patch. In MAE's ablation (ViT-L, 800 epochs), normalized targets gave 85.4 percent after fine-tuning and 73.9 under a linear probe, against 84.9 and 73.5 for raw pixels.

One more choice is worth carrying into an interview. MAE takes the loss on hidden patches only, like BERT. A classical denoising autoencoder grades the reconstruction of every patch, and in the paper doing so cost about 0.5 points of accuracy.

import numpy as np

def mae_loss(pred, imgs, mask, P, norm_pix=True, eps=1e-6):
    # Placeholder: plain MSE over every patch against raw pixels. Fix it.
    return float(((pred - patchify(imgs, P)) ** 2).mean())

One forward pass with real shapes

Here is the whole pipeline once, in NumPy with random weights, before we count its cost. The tokens come from a $224\times224$ RGB image with 16-pixel patches, the real input size. The widths are small so that the cell runs quickly. The encoder is 192 wide, the width of the smallest standard ViT (ViT-Ti), and the decoder is 128 wide. Each side has one Transformer block with a single attention head.

Follow the shapes. The encoder block runs on a $50\times192$ matrix, the decoder block on $197\times128$, and the head produces $196$ rows of $768$ values, one per patch. The $147$ hidden rows are graded. The loss at random weights is 1.97, far above the trivial 0.9987, because the random head outputs values of size about one. Training would first drive the loss down to the trivial level and then below it.

What the asymmetric encoder saves

Dropping the hidden patches from the encoder is what lets MAE pretrain large models cheaply, and the saving can be computed exactly. We count FLOPs, floating-point operations, with each multiply-add counted as two. Recall the per-block cost from §5.6. A Transformer block reading $T$ tokens of width $w$, with an MLP of width $4w$, costs $24Tw^2$ forward FLOPs in its weight matrices. That is $6Tw^2$ for the query, key, and value projections, $2Tw^2$ for the output projection, and $16Tw^2$ for the MLP. The two attention products, scores and the weighted sum, add $4T^2w$. §27.2 counts the same quantities in multiply-adds, which halves every number.

Proof. Divide the numerator and the denominator by $24N_pLd^2$ and use $T_v/N_p = 1 - \rho$. The derivative of $(1+r)/(1-\rho+r)$ with respect to $\rho$ is $(1+r)/(1-\rho+r)^2 \gt 0$. Cross-multiplying, $(1+r)/(1-\rho+r) \le 1/(1-\rho)$ is the same as $r(1-\rho) \le r$, which holds because $r \ge 0$, with equality for $\rho \gt 0$ only when $r = 0$. As $\rho \to 1$ the ratio tends to $(1+r)/r$. ∎

In plain words, the encoder's share of the cost shrinks to $1 - \rho$ of what it was, and the decoder's share does not shrink at all. So only a light decoder lets the saving approach $1/(1-\rho)$.

The cell adds the attention term and the class token, one at a time.

With attention included and no class token the count gives 3.29, the 3.3 times that MAE reports. The class token and attention together give 3.25. Adding the patch embedding, the decoder's input projection, and the prediction head, as Exercise 28.5d does, brings it to 3.22. The heavier ViT-H/14 encoder saves more, 3.61 times, because the same decoder is a smaller share of its cost.

Measured time follows the count.

def mae_flops(L, d, L_dec, d_dec, N_p, rho, P=16, C=3, cls=True):
    raise NotImplementedError

Why the encoder never sees a mask token

Saving compute is half the reason. The other half is a mismatch between pretraining and use. An encoder trained with mask tokens sees inputs that are 75 percent mask tokens during pretraining and 0 percent afterwards, when it reads whole images. BERT has a milder version of the same problem at 15 percent, which is why it replaces only 80 percent of its selected tokens with [MASK] (§7.1). MAE's ablation measured the cost. Putting mask tokens into the ViT-L encoder lowered the linear probe from 73.5 to 59.6 percent, the "14 percent" drop the paper reports, and fine-tuning from 84.9 to 84.2, while tripling the compute.

Some architectures cannot drop tokens. A Swin Transformer (§27.5) computes attention inside windows of a regular grid and merges neighboring patches between stages. A hole in the grid breaks both operations. SimMIM, designed with Swin in mind, therefore keeps mask tokens in the encoder and pays the full compute. ConvNeXt V2 (Woo et al. 2023) found a third way for convolutional networks. During pretraining it treats the visible patches as a sparse input and runs sparse convolutions, which compute only at the visible positions and skip the hidden ones.

An interviewer will often ask which of the two reasons matters more. For a frozen encoder it is the mismatch, 14 points of linear probing. For the training bill it is the compute, about three times.

What MAE's ablations show

MAE's ablation table is among the most quoted in self-supervised vision, and each row answers a design question you may be asked. Fine-tuning and the linear probe are the two tests from the opening.

The decoder rows explain a lot. With a one-block decoder, the encoder's last layers must do part of the reconstruction themselves, so they specialize in pixels. That hurts a frozen linear probe (65.5) but not fine-tuning (84.8), where those layers are retrained. A deeper decoder takes over the pixel-specific work and leaves the encoder's output more abstract.

The augmentation rows are the opposite of contrastive learning. MAE works with no augmentation at all, because a fresh random mask every epoch already makes each example new, and color jitter hurts it. Methods that learn by matching two augmented views depend on augmentation heavily. Restricted to crops alone, the contrastive SimCLR (§28.2) loses 28 points and BYOL, which predicts a slowly moving copy of itself (§28.3), loses 13, as MAE notes citing Grill et al. (2020). The practical lesson is that masking is MAE's augmentation, so you do not need to tune an augmentation policy for it.

A linear probe at 76.6 and fine-tuning at 86.9 for the same network is a ten-point disagreement between two tests of one representation. We come back to what it means at the end of the section.

In PyTorch

The block below trains a complete MAE on $16\times16$ random fields with 4-pixel patches. The encoder has two blocks and the decoder one, built from nn.TransformerEncoderLayer. It masks with torch.argsort and restores the order with torch.gather, exactly as MAE's models_mae.py does. With only 16 values per patch, predicting zeros already scores $(n-1)/n = 0.9375$, and a model that has learned nothing sits there. The sine-cosine positions use base 100 instead of 10,000. On a $4\times4$ grid the slow frequencies of base 10,000 barely change from one patch to the next, so they would barely tell the patches apart.

The loss stays near the trivial 0.94 for about the first 50 steps. Then it falls below it as the model starts to use the visible patches, to about 0.88 on held-out images after 150 steps. Real MAE pretraining shows the same shape on a far longer scale. In practice you would not write the model yourself. Hugging Face's ViTMAEForPreTraining implements the paper, with the released checkpoints at facebook/vit-mae-base, -large, and -huge.

Predicting a discrete code for each patch

MAE was not the first masked image model for Vision Transformers. BEiT (Bao et al. 2022) came a few months earlier and took BERT more literally. BERT predicts each hidden word from a fixed vocabulary with a softmax, so BEiT first turns each patch into a word. An image tokenizer is a network trained beforehand to compress an image into a grid of discrete codes, one integer per patch from a fixed codebook, and to rebuild the image from them. BEiT borrows the tokenizer of the DALL-E image generator, a discrete variational autoencoder (dVAE), an autoencoder whose latent is a grid of such codes.

In plain words, BEiT asks which of 8,192 visual words each hidden patch was, according to a tokenizer trained beforehand to compress images. How such tokenizers are built, with a discrete bottleneck trained to reconstruct the image, is the subject of discrete codes and image tokenizers.

BEiT hides patches in blocks, and the reason is the isolation count from earlier. At 40 percent random masking the theorem gives an isolation probability of only about 0.024, so almost every hidden patch has a visible neighbor. Blocks fix that without raising the ratio, because the inside of a block is far from anything visible.

BEiT's blockwise masking works like this. It draws a block area $s$ of at least 16 patches, an aspect ratio $r$ between 0.3 and $1/0.3$, and a position. It hides the rectangle of about $\sqrt{sr}$ rows and $\sqrt{s/r}$ columns. It repeats until 40 percent of the grid is hidden. The released code draws $\log r$ uniformly, so tall and wide blocks are equally likely, and it never hides more than 75 of the 196 patches.

BEiT's ablation found that blocks helped, more for segmentation than for classification, and more still when the target was pixels, the case where copying neighbors is most tempting. MAE's table shows the other side. At 75 percent, random masking is already hard enough, and blocks make it too hard (82.8 / 63.9 against 84.9 / 73.5).

import numpy as np

def blockwise_mask(h, w, rng, ratio=0.4, min_patches=16, min_aspect=0.3, max_attempts=1000):
    """Return (mask of shape (h, w), dtype bool, True = masked) and the list of placed blocks (top, left, a, b)."""
    raise NotImplementedError

The smallest version that works

SimMIM (Xie et al. 2022) asked which of BEiT's pieces were needed, and kept almost none of them.

In plain words, hide big squares, keep a placeholder where they were, and read their raw colors back out with one linear layer.

With ViT-B, SimMIM reached 83.8 percent fine-tuned on ImageNet-1k, above BEiT's 83.2, with no tokenizer, no blocks, and a linear head. Heavier prediction heads reconstructed the pixels slightly better and transferred no better. Better inpainting is not the goal. Capacity spent in the head is capacity the encoder does not have to learn.

SimMIM and MAE agree on the essentials, which are random masking with large enough holes, pixel targets, and a light head. They disagree on the encoder's input. MAE drops the hidden patches and trains a ViT about three times faster. SimMIM keeps them and works with any backbone.

Choosing what to predict

Every design so far predicts something computed from the pixels of the hidden patch alone. The choice of target decides what the encoder must represent, and the options form a ladder from low level to high level.

Histograms of oriented gradients (HOG) are a hand-made image feature. They take the image's gradients, which removes brightness. They count the gradient directions in each $8\times8$-pixel cell, and they normalize each histogram, which removes contrast. MaskFeat (Wei et al. 2022) predicts the HOG features of the hidden regions. That local contrast normalization was essential in its ablation, the same lesson as MAE's per-patch normalization. With 40 percent blockwise masking, mask tokens in the encoder, and a squared-error loss, MaskFeat reached 84.0 percent with ViT-B and 85.7 with ViT-L on ImageNet-1k.

The next two methods make the target a network's output, and they inherit DINO's machinery (§28.4). Here are its pieces again. A student network with parameters $\theta$ is trained by gradient descent. A teacher with the same architecture has parameters $\xi$ that follow an exponential moving average (EMA) of the student, $\xi \leftarrow m\xi + (1-m)\theta$ after each step, with momentum $m$ close to 1. The teacher receives no gradient. Because $m$ is close to 1 it changes slowly, and §28.2 showed that it averages roughly the last $1/(1-m)$ students.

In plain words, the teacher looks at the whole view and says what each patch is, as a distribution over 8,192 learned prototypes. The student sees the view with holes and must say the same about the holes. The teacher plays the part of BEiT's tokenizer, but it is trained along with the student, which is why iBOT (Zhou et al. 2022) calls it an online tokenizer. The class-token loss keeps its prototypes meaningful. Soft distributions worked better than one-hot codes, since a patch rarely belongs to exactly one visual word.

iBOT's frozen features are far stronger than MAE's. With ViT-B/16 on ImageNet-1k it reached 77.1 percent with a weighted $k$-nearest-neighbor vote, which labels an image by the labels of its most similar training images in feature space (§28.1). It reached 79.5 with a linear probe, and 84.0 fine-tuned. Pretrained with ViT-L/16 on ImageNet-22k, a larger collection of about 14 million images, it reached 82.3 linear and 86.6 fine-tuned at 224 pixels, and 87.8 at 512.

In plain words, predict what the teacher's top layers say about a patch when the teacher sees everything. Unlike pixels, codes, or HOG, this target depends on the whole image. The same patch of fur has a different target in a cat than in a rug. data2vec (Baevski et al. 2022) argues this is why it beat the local targets. Normalizing each block's output matters for the reason §28.3 taught, since without it a constant representation would predict itself perfectly. data2vec reached 84.2 percent fine-tuned with ViT-B and 86.6 with ViT-L.

A further step replaces the self-trained teacher with an encoder that learned from language. CLIP (§30.2) trains an image encoder and a text encoder so that matching images and captions get similar vectors. BEiT v2 (Peng et al. 2022) trains its tokenizer's codes to reconstruct CLIP's features instead of pixels. EVA (Fang et al. 2023) regresses CLIP's features of the hidden patches directly. Both bring meaning learned from image-text pairs into masked modeling, at the price of depending on another model.

Masked prediction also proved its worth inside the joint-embedding methods of this chapter, the ones that compare whole images. The question there is whether a loss on each patch adds anything to a loss on the whole image.

Each column is one design choice, from what is hidden to the loss, and each method is one path through the columns. Click a method to see its path, or hover a box to see which methods share that choice and, for a target, what makes it. Most of the methods differ mainly in what they predict and in what makes the target. The strip below the map gives each method's ImageNet accuracy with ViT-B, and the choices barely move fine-tuning while they move the frozen features a lot.

Predicting the next patch instead of a hidden one

There is a second way to borrow from language. A language model like GPT predicts each token from the tokens before it, with no mask at all, and an image can be read the same way in raster order.

In plain words, an autoregressive model is the extreme of masked modeling, in which every patch is hidden from all the patches before it.

Its appeal is scaling. iGPT-L, with 1.4 billion parameters, reached 65.2 percent under linear probing on $48\times48$ inputs. A larger iGPT-XL, with 6.8 billion parameters, reached 68.7 from its best single layer and 72.0 with features concatenated from five layers (as tabulated by BEiT). AIM trained models from 0.6 to 6.5 billion parameters (named AIM-0.6B to AIM-7B) on about 2 billion web images, with some ImageNet mixed in. Downstream accuracy rose with model size and tracked the pretraining loss, with no sign of saturation at the largest size. AIM-7B reached 84.0 percent on ImageNet-1k with a frozen trunk and an attentive probe, a single learned cross-attention layer that reads all the patch tokens (§28.6). AIMv2 (Fini et al. 2024) adds a decoder that generates the image patches and then the caption's words, and its 3-billion-parameter encoder reached 89.5 percent frozen at 448 pixels. The same masked and next-token prediction over image codes also drives masked and autoregressive image generation.

Where masked modeling falls short

Masked image models are excellent starting points for fine-tuning and mediocre frozen feature extractors. DINOv2's evaluation puts numbers on it. MAE's ViT-H/14 scores 49.4 percent with a weighted $k$-nearest-neighbor vote and 76.6 with a linear probe on ImageNet-1k. DINOv2's ViT-g/14, trained by self-distillation with a masked-patch loss, scores 83.5 and 86.5 (Oquab et al. 2023, Table 4). Yet the same MAE model fine-tunes to 86.9. Three explanations fit together.

The objective keeps low-level detail. Reconstructing pixels rewards an encoder that preserves texture, color, and exact position, and a classifier needs little of that. The decoder ablation showed the last encoder layers specializing in reconstruction when the decoder is shallow. Even with the default decoder some of that remains, so a linear map from the last layer has to work through the detail. For ViT-L, fine-tuning only the last block, which adds a little nonlinearity, recovers most of the gap (73.5 to 81.0).

Contrastive and masked features differ in kind. Park et al. (2023) compared the two families on Vision Transformers. Contrastive learning trains attention to capture global patterns such as an object's shape, mostly in the later layers, and that is what makes images linearly separable in its features. But its attention maps become nearly the same for every query and every head. It also relies on low-frequency signals, which carry shape. Masked modeling relies on high frequencies, which carry texture. It shapes mainly the early layers and keeps attention diverse, which helps dense prediction and fine-tuning.

Masking is the only invariance. MAE uses crops alone, so it learns no invariance to color. That is a loss for ImageNet's classes and a gain for tasks where color and fine detail matter. Web-SSL (Fan et al. 2025) trained DINO models of 1 to 7 billion parameters and MAE models of 1 to 5 billion on 2 billion web images. It tested each as the image encoder of a vision-language model that answers questions about images. Web-MAE beat Web-DINO on questions that need reading text (OCR, optical character recognition) and charts at every size, and lost on the other question types.

The fix the field converged on is to combine the two. iBOT adds a masked-patch loss to DINO, and DINOv2 and DINOv3 keep it. As of late 2026 the result is a single encoder that is strong frozen for both whole-image and dense tasks. Pure MAE survives as a starting point. It is the pretraining of Segment Anything's image encoder (§27.6), and in video VideoMAE hides 90 percent of space-time tubes.

All rows are pretrained on ImageNet-1k (BEiT's tokenizer aside) and report top-1 accuracy in percent from the papers above. The fine-tuning numbers span one point, and the costs do not. MAE's encoder reads a quarter of the tokens and needs no second network. The teacher methods run a full-image teacher pass at every step.

The next step predicts a representation of the hidden regions instead of their pixels, and does it without hand-made augmentations. That is the step taken by I-JEPA (Assran et al. 2023). It predicts the features of large target blocks from a single context block, and it trains a ViT-H/14 on ImageNet in under 72 hours on 16 A100 GPUs.

Read more

• He et al. (2022). Masked Autoencoders Are Scalable Vision Learners. The MAE paper. Section 3 for the design, Table 1 for the ablations quoted here, Figure 9 for partial fine-tuning, and Appendix A for the recipes.
• Bao, Dong, Piao & Wei (2022). BEiT: BERT Pre-Training of Image Transformers. Masked prediction of dVAE codes, with Algorithm 1 for blockwise masking.
• Xie et al. (2022). SimMIM: A Simple Framework for Masked Image Modeling. Strips masked modeling to random masking, raw pixels, and a linear head, and introduces AvgDist.
• Wei et al. (2022). Masked Feature Prediction for Self-Supervised Visual Pre-Training. MaskFeat compares five targets and finds HOG, with its contrast normalization, works best for the cost.
• Zhou et al. (2022). iBOT: Image BERT Pre-Training with Online Tokenizer. Masked prediction with an EMA teacher as the tokenizer, the patch loss DINOv2 and DINOv3 kept.
• Baevski et al. (2022). data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. Regressing averaged, normalized teacher layers, with one recipe for three modalities.
• El-Nouby et al. (2024). Scalable Pre-training of Large Autoregressive Image Models. AIM, next-patch prediction with prefix attention, scaled to 7 billion parameters.
• Park et al. (2023). What Do Self-Supervised Vision Transformers Learn?. Why contrastive and masked features differ, through attention, frequency, and depth.

Watch next

▶

▶