How a network that only learns to copy its input becomes a generative model, and how images become the discrete tokens and compact latents that modern image and video generators are built on. Autoencoders, the ELBO, VAEs, posterior collapse, vector quantization, and image tokenizers, built from first principles.

An autoencoder is a neural network trained to copy its input through a narrow middle layer. Its first half, the encoder, turns each input into a short list of numbers called the code. Its second half, the decoder, rebuilds the input from the code alone. Because the code is short, the network cannot copy everything, so it has to learn what in the input is worth keeping.

Most modern image and video generators are built on this idea. A 512 by 512 color photograph is $512 \times 512 \times 3 = 786{,}432$ numbers. The autoencoder inside Stable Diffusion, a widely used open text-to-image model, turns it into a grid of $64 \times 64 \times 4 = 16{,}384$ numbers, 48 times fewer. From that grid alone it rebuilds the photograph with a typical error of about 6 percent of the pixel range (Rombach et al. 2022). The generator then works on the small grid and never touches the pixels. This is possible because natural images do not fill the space of all pixel arrays. They lie near a much smaller surface, and the autoencoder learns coordinates on that surface.

Until now the book has taught models to read images. Vision Transformers cut an image into patches and turn each patch into a vector, CLIP puts images and captions into one space, and vision-language models answer questions about a picture. This part of the book goes the other way and teaches models to make images. The models of §31.6 that write images already work this way. They produce a short code like Stable Diffusion's grid, and a decoder turns it back into pixels. This chapter builds those codes and decoders, the next one builds the diffusion models that generate in them, and Chapter 34 scales both to text-to-image and video.

This section builds autoencoders up from the simplest case. When both halves are linear, an autoencoder learns the same subspace as principal component analysis, though not the same axes, and a little weight decay recovers the axes too. Any autoencoder trained with squared error converges to an average of the inputs it cannot tell apart. That average explains blurry outputs, and it leads directly to diffusion models. Then come the variants used today, the autoencoder of a real image generator, and its use as an anomaly detector. The section ends with the reason a plain autoencoder cannot make new images, the failure that motivates the rest of this chapter.

Copying through a narrow middle layer

Start with a picture you can draw. Take points in the plane that lie near a line through the origin. Write each one as $\mathbf{x} = t\,\mathbf{u} + \text{noise}$, where $\mathbf{u}$ is a unit vector along the line and $t$ is a random number. Each point takes two numbers to write down, but the single number $t$ describes it almost perfectly.

Suppose you must send each point to a friend as one number. You would send $z = \mathbf{u}^\top\mathbf{x}$, the point's position along the line. Your friend would rebuild $\hat{\mathbf{x}} = \mathbf{u}\,z$. All that is lost is the noise perpendicular to the line.

An autoencoder learns both halves of this protocol from data. The encoder maps a point to a short code, and the decoder maps the code back. Training makes the reconstruction as close to the input as it can. Nobody tells the network the direction $\mathbf{u}$. It finds $\mathbf{u}$ because every other direction loses more.

In plain words, encode each example, decode it, measure how far the result is from the original, and average over the data. In this chapter $\phi$ always names the encoder's parameters and $\theta$ the decoder's. The narrow middle layer is called the bottleneck, and its width is the code dimension $d_z$.

The bottleneck is what forces the network to learn. Without one, an overcomplete autoencoder with enough capacity can make the decoder undo the encoder exactly. It reaches zero loss and has learned nothing about the data. So something must make copying hard, and a code shorter than the input, $d_z < d_x$, is the simplest way. Later in this section we meet four other ways, which are noise, sparsity, contraction, and masking. Each decides differently which features of the input survive.

The cell draws 20,000 points from a two-dimensional Gaussian and sends them through a one-number code along four directions. The encoder takes the coordinate along a unit vector $\mathbf{v}$, and the decoder multiplies that coordinate back by $\mathbf{v}$.

The code along 22.5 degrees loses about 0.587 per point, and every other direction loses more. Here is why, worked by hand.

What this changes in practice is a habit of thought. A short code keeps the directions where the data vary most, and its error is the variance it drops. An interviewer who asks what a one-number bottleneck keeps wants exactly that answer. The rest of this part proves that a linear autoencoder learns this in any dimension.

A linear autoencoder learns the principal subspace

First, a short recap of principal component analysis (PCA) from §0.10. PCA finds the directions along which data vary most. For data with mean zero, those directions are the eigenvectors of the covariance matrix, and the variance along each one is its eigenvalue. Keeping the top $k$ directions and dropping the rest gives the best rank-$k$ approximation of the data, a result called the Eckart-Young theorem (§0.10). Its squared error is the sum of the dropped eigenvalues. §1.6 used PCA to draw pictures of embedding spaces. The new question is whether a network that only learns to copy, and is never told about eigenvectors, finds the same answer.

Make both maps linear and drop the biases for now. The encoder is a matrix $\mathbf{W}_1 \in \R^{d_z \times d_x}$ and the decoder a matrix $\mathbf{W}_2 \in \R^{d_x \times d_z}$. The reconstruction is $\hat{\mathbf{x}} = \mathbf{W}_2\mathbf{W}_1\mathbf{x}$. Assume the data have mean zero (a later subsection removes this assumption) and covariance

Here $\mathbf{U} = [\mathbf{u}_1, \dots, \mathbf{u}_{d_x}]$ is an orthogonal matrix whose columns are the eigenvectors, called the principal directions. The diagonal matrix $\boldsymbol\Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_{d_x})$ holds the eigenvalues in descending order, $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_{d_x} \ge 0$. In this section $\lambda_i$ is always an eigenvalue, never a weight decay. The expectation can be over the data distribution or over the $N$ examples with weight $1/N$ each, and everything below holds for either. Write $\mathbf{U}_{d_z} \in \R^{d_x \times d_z}$ for the first $d_z$ columns of $\mathbf{U}$.

Two more terms are needed. The column space of the decoder is the set of all its possible outputs $\mathbf{W}_2\mathbf{z}$, a subspace of dimension at most $d_z$. The orthogonal projector onto a subspace is the matrix that drops each point perpendicularly onto it, landing on the closest point of the subspace.

In plain words, the best a linear copier can do is keep the $d_z$ directions of largest variance and lose the rest. Its error is exactly the variance it loses, the sum of the dropped eigenvalues, which we call the tail. It is free to describe the kept subspace in any basis it likes, and that freedom is what the matrix $\mathbf{G}$ expresses. This is a theorem, proved by Bourlard and Kamp (1988) and by Baldi and Hornik (1989).

Baldi and Hornik also studied the rest of the loss surface. A critical point is a set of weights where the gradient is zero. A saddle point is a critical point where the loss still decreases in some direction. They showed that, when the eigenvalues are distinct, every critical point other than the global minima is a saddle. So gradient descent has no bad local minimum to get stuck in.

The proof rests on one observation. Whatever the encoder does, the output lies in the decoder's column space, so no output can beat the orthogonal projection of $\mathbf{x}$ onto that space. The question then becomes which subspace of dimension $d_z$ holds the most variance. The answer is the span of the top eigenvectors.

Notice what the proof never used. It never assumed the data are Gaussian, only that they have a covariance. And it never asked how the network was trained, only what the best possible weights are.

Centering, and the bias terms

Real autoencoders have biases, $\hat{\mathbf{x}} = \mathbf{W}_2(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2$. The two biases only add a constant vector $\mathbf{c} = \mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2$ to the output. For fixed weights, the best constant is the average of what the weights fail to rebuild, $\mathbf{c} = (\mathbf{I} - \mathbf{W}_2\mathbf{W}_1)\boldsymbol\mu$, where $\boldsymbol\mu$ is the data mean. With that constant, the residual becomes $(\mathbf{I} - \mathbf{W}_2\mathbf{W}_1)(\mathbf{x} - \boldsymbol\mu)$. That is the same problem as before, on the centered data $\mathbf{x} - \boldsymbol\mu$ (Bourlard and Kamp 1988). So a linear autoencoder with biases learns the principal subspace of the centered data, exactly as PCA does after subtracting the mean.

Without biases and without centering, the theorem still applies, but to the wrong matrix. It sees the second moment $\mathbb{E}[\mathbf{x}\mathbf{x}^\top] = \boldsymbol\Sigma_x + \boldsymbol\mu\boldsymbol\mu^\top$, not the covariance. When the mean is large, the top eigenvector of that matrix points almost along $\boldsymbol\mu$. The autoencoder then spends one of its few code dimensions describing the mean, which never changes. In practice, always center before comparing with PCA, and expect an interviewer to ask why. Exercise 32.1a has a test that catches exactly this mistake.

The subspace, not the components

The last line of the theorem holds a twist. Suppose $(\mathbf{W}_1, \mathbf{W}_2)$ is optimal. Then so is $(\mathbf{G}^{-1}\mathbf{W}_1, \mathbf{W}_2\mathbf{G})$ for every invertible $\mathbf{G} \in \R^{d_z\times d_z}$. The product $\mathbf{W}_2\mathbf{G}\mathbf{G}^{-1}\mathbf{W}_1$ is unchanged, and the loss only sees the product.

So a linear autoencoder learns the principal subspace, the plane or higher-dimensional slice that the top directions span. It does not learn the principal components, the individual directions themselves. Its decoder's columns are some basis of the subspace. They are generally not orthogonal, not of unit length, and not sorted by variance, and the code coordinates are correlated with each other.

The widget trains a linear autoencoder with a two-number code on two-dimensional Gaussian data. The dashed lines are the principal axes and the arrows are the decoder's columns. The solid curve is where the decoder sends the circle of unit-length codes. The slider $\alpha$ adds weight decay, which a later part of this section explains.

With $\alpha = 0$ the curve settles into a tilted ellipse that changes with every initialization, because every basis of the plane is optimal. How does gradient descent pick one basis out of all the optimal ones? It depends on where training starts, and on one quantity that training leaves unchanged.

Training one by gradient descent

The cell trains a linear autoencoder by full-batch gradient descent on 2,000 points in 10 dimensions, centered first. It compares the result with PCA computed by an eigendecomposition. For that it needs a way to say how close two subspaces are, and the tool is the set of principal angles between them. If $\mathbf{Q}_a$ and $\mathbf{Q}_b$ hold orthonormal bases of two subspaces of dimension $d_z$, the singular values of $\mathbf{Q}_a^\top\mathbf{Q}_b$ are the cosines of the principal angles. The subspaces coincide exactly when every angle is zero.

Read the output in three parts. First, the loss falls to the tail, 1.646, which is the best any linear code can do. Second, the largest principal angle to the PCA subspace shrinks toward zero, so the autoencoder found the right subspace. Third, the three columns are still not the principal directions. They are nearly orthonormal, with cosines of about 0.02 between them and singular values near 1, as the balance argument predicts. But their cosines with $\mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3$ are only about 0.36, 0.90, and 0.43. The three singular values are nearly equal, so even the singular value decomposition of $\mathbf{W}_2$ cannot tell the components apart.

import numpy as np

def train_linear_ae(X, d_z, steps, lr, seed=0):
    raise NotImplementedError

Recovering the components from a trained autoencoder

Suppose a colleague hands you a trained linear decoder $\mathbf{W}_2$. Its columns span the principal subspace but are mixed by an unknown $\mathbf{G}$. You can still recover the individual components. You only need a small $d_z \times d_z$ eigenproblem, never the full $d_x \times d_x$ covariance. The idea is to rotate inside the subspace until the code coordinates become uncorrelated.

1. Orthonormalize the columns with a QR factorization, $\mathbf{W}_2 = \mathbf{Q}\mathbf{R}$. The matrix $\mathbf{Q} \in \R^{d_x\times d_z}$ has orthonormal columns spanning the same subspace.
2. Project the centered data onto those columns. The coordinates have a small covariance $\mathbf{C}_z = \mathbf{Q}^\top\boldsymbol\Sigma_x\mathbf{Q} \in \R^{d_z \times d_z}$.
3. Diagonalize $\mathbf{C}_z$ and rotate its eigenvectors back with $\mathbf{Q}$.

Why it works. If the subspace is $\operatorname{span}(\mathbf{u}_1, \dots, \mathbf{u}_{d_z})$, then $\mathbf{Q} = \mathbf{U}_{d_z}\mathbf{O}$ for some orthogonal $\mathbf{O}$. So $\mathbf{C}_z = \mathbf{O}^\top\operatorname{diag}(\lambda_1, \dots, \lambda_{d_z})\mathbf{O}$. Its eigenvectors are the columns of $\mathbf{O}^\top$, and its eigenvalues are $\lambda_1, \dots, \lambda_{d_z}$. Rotating back gives $\mathbf{Q}\mathbf{O}^\top = \mathbf{U}_{d_z}$. Numerical linear algebra calls this the Rayleigh-Ritz procedure. It turns any method that finds the subspace into PCA, which is the follow-up an interviewer asks once you say "subspace, not components".

A shortcut that is often proposed, taking the left singular vectors of $\mathbf{W}_2$, does not work on its own, as the training cell above showed. Plaut (2018) found that it worked in practice. Kunin et al. (2019) traced that to the weight decay in his training code, which the next part of this section analyzes.

import numpy as np

def components_from_ae(W2, X):
    norms = np.linalg.norm(W2, axis=0)
    return W2 / norms, norms ** 2          # placeholder: just normalizes the columns

Weight decay picks out the principal directions

There is a second way to recover the components, and it comes almost for free. Weight decay adds a cost proportional to the squared size of the weights. Added to the loss, it reads

where $\lVert\cdot\rVert_F$ is the Frobenius norm, the square root of the sum of squared entries. We write $\alpha$ for the strength because $\lambda_i$ already names the eigenvalues. Strictly, this is an $L_2$ penalty added to the loss. For plain gradient descent the penalty and weight decay are the same. For Adam they differ, as §0.6 explained, and the theorem below is about the penalty.

The reconstruction term still only sees the product $\mathbf{W}_2\mathbf{W}_1$, but the penalty sees the two factors. Among all the ways to write the same product, it prefers the cheapest. That preference is enough to pin down the components.

The theorem uses the singular value decomposition of §0.10. It writes any matrix as a rotation, then a stretch of each axis by a singular value, then another rotation. The left singular vectors are the output directions that get stretched.

In plain words, weight decay makes the encoder the transpose of the decoder. It shrinks each kept direction by its own factor $1 - \alpha/\lambda_i$, strong directions a little and weak ones a lot. It switches off any direction whose variance is below $\alpha$. Because the shrink factors differ from direction to direction, mixing the directions would now change the loss. What stays free is the orthogonal $\mathbf{O}$ on the code side. So the columns of $\mathbf{W}_2$ can still be rotated, but its left singular vectors cannot.

Check it with numbers. Take $\lambda_1 = 4$ and $\alpha = 1$. The first direction is kept with $m_1 = 1 - 1/4 = 0.75$, so the decoder's singular value is $\sqrt{0.75} \approx 0.866$. Reading it back, $\alpha/(1 - 0.75) = 4$ recovers the eigenvalue. A direction with $\lambda = 0.8$ is switched off, since $1 - 1/0.8$ is negative.

Kunin et al. (2019) proved this theorem. They also showed that every critical point is symmetric, $\mathbf{W}_1 = \mathbf{W}_2^\top$. And they related the minimum to the most probable parameters of probabilistic PCA, the model §32.2 builds. Here is the argument in three steps, with the one technical step left to the paper.

The cell repeats the 10-dimensional experiment with $\alpha = 0.5$.

Every prediction of the theorem holds to the printed precision. The singular values are $\sqrt{1 - \alpha/\lambda_i}$. The left singular vectors are the principal directions, with cosines of 1. The eigenvalues come back exactly from the singular values. The encoder is the transpose of the decoder, and the loss equals $L_\alpha^\star$.

So weight decay is more than a regularizer here. It is the ingredient that turns a subspace learner into a PCA algorithm. An interviewer who asks "does a linear autoencoder learn PCA?" wants both halves of the answer, the subspace without weight decay and the components with it.

import numpy as np

def regularized_lae_optimum(Sigma, d_z, alpha):
    raise NotImplementedError

def pca_from_decoder(W2, alpha):
    raise NotImplementedError

Nonlinear autoencoders find curved surfaces

A linear autoencoder can only keep a flat subspace. Most data lie near curved surfaces. There the right code is a coordinate along the curve, and no projection can compute it. Make the encoder and decoder small neural networks (§0.7), and the same reconstruction loss can learn such coordinates.

This is the result that made deep autoencoders famous. Hinton and Salakhutdinov (2006) trained an autoencoder on MNIST handwritten digits, with layers of 784, 1000, 500, 250, and 30 units. Its 30-number codes rebuilt test digits with an average squared error of 3.00 per image, against 13.87 for 30-component PCA. They had to initialize the network one layer at a time with restricted Boltzmann machines, small probabilistic models trained layer by layer. Without that start, their deepest network, trained on synthetic images of curves, only learned to output the average of its training images. Modern initialization and normalization have made the layer-by-layer start unnecessary, but the comparison with PCA stands.

Here is the smallest version that still makes the point. The data are 600 points near four arcs of the unit circle, with radial noise of standard deviation 0.04, and the code is a single number. The best linear one-number code only halves the error of sending nothing, because the cloud is nearly round and every direction holds about half its variance. The first cell builds the network, with one hidden layer of 32 tanh units in each half. It writes the backward pass by hand with the chain rule of §0.2. The second cell trains the network with Adam, the optimizer that gives every parameter its own step size (§0.6).

Convolutional autoencoders, and the downsampling factor

For images, the encoder is a stack of strided convolutions, and the decoder mirrors it with layers that upsample. Each stride-2 convolution halves the height and width. By the output-size formula of §0.8, a $3\times3$ kernel with padding 1 and stride 2 maps a side of $n$ to $\lfloor (n - 1)/2\rfloor + 1$, which is $n/2$ for even $n$. The decoder uses transposed convolutions, the upsampling layers that run a strided convolution's shape change in reverse. One with kernel 4, stride 2, and padding 1 doubles the side, since its output size is $(n - 1)\cdot 2 - 2 + 4 = 2n$.

Three halvings give a downsampling factor $f = 8$, meaning the latent grid is 8 times smaller per side. A latent with $c$ channels on that grid holds $c/(3f^2)$ as many numbers as an RGB image. For Stable Diffusion's $f = 8$ and $c = 4$ that is $4/192 = 1/48$, the ratio from the opening of this section.

The block below shows the production pattern on small synthetic images of bars. It trains a denoising autoencoder, which gets a noisy input and a clean target. The next part of this section explains why that helps.

On a CPU run the $16\times16$ image becomes a $4\times4\times4$ code, 4 times fewer numbers. The per-pixel error drops from about 0.090 for the noisy input to about 0.010 for the output, so after 300 steps the network already removes most of the noise.

The best reconstruction is a conditional mean

So far the input and the target were the same. Now corrupt the input. Feed the network a damaged copy $\tilde{\mathbf{x}}$ and ask it to rebuild the clean $\mathbf{x}$. What is the best function it could possibly learn? The answer does not depend on the network at all, and it is one of the most useful facts in this book.

Start with a tiny case. Suppose one pixel is either 0 (black) or 1 (white), each with probability one half, and the damaged copy says nothing about which. If the network outputs a value $g$, its average squared error is $\tfrac12 g^2 + \tfrac12(1 - g)^2$. Outputting 0 or 1 costs 0.5. Outputting 0.5 costs $\tfrac12 \cdot 0.25 + \tfrac12 \cdot 0.25 = 0.25$, which is the minimum. The best guess is the average of the possibilities, a gray that never occurs in the data.

The general rule replaces "the average of the possibilities" with the conditional mean $\mathbb{E}[\mathbf{x} \mid \tilde{\mathbf{x}}]$. It is the average of all the clean inputs that could have produced the damaged copy $\tilde{\mathbf{x}}$, each weighted by how likely it is.

The theorem covers every case in this chapter. With $\tilde{\mathbf{x}} = \mathbf{x}$ it is trivial. With $\tilde{\mathbf{x}}$ a noisy or masked copy, it says what a denoiser learns. It also applies to a plain autoencoder's decoder, with the code $\mathbf{z}$ in the role of $\tilde{\mathbf{x}}$. The best decoder outputs $\mathbb{E}[\mathbf{x}\mid\mathbf{z}]$, the average of all inputs that share a code.

What it changes in practice is how you read blur. When the code does not pin down the input, the best output is a blend. A blend of sharp images is a blurry image, just as the blend of black and white was gray. This is the root of the blur that §32.4 analyzes in VAEs, and the reason §32.6 adds losses other than squared error. "Why are autoencoder reconstructions blurry?" is a standard interview question, and this theorem is the answer.

The corruption makes copying useless, because a copy returns the noise. So a denoising autoencoder needs no bottleneck at all. It is forced to learn what clean data look like, which is exactly the knowledge a generator needs. To see how much it learns, work out the one case where the conditional mean has a closed form.

There is a second way to write the same denoiser, and it is the bridge to diffusion models. Let $p_\sigma$ be the density of the noisy $\tilde{\mathbf{x}}$, here $\mathcal{N}(\mathbf{0}, \boldsymbol\Sigma_x + \sigma^2\mathbf{I})$. Its score is the gradient of its log density, which points in the direction where noisy data become more likely. For this Gaussian, $\nabla_{\tilde{\mathbf{x}}}\log p_\sigma(\tilde{\mathbf{x}}) = -(\boldsymbol\Sigma_x + \sigma^2\mathbf{I})^{-1}\tilde{\mathbf{x}}$. Then

In plain words, the best denoiser takes the noisy point and steps uphill on the log density of noisy data, by $\sigma^2$ times the gradient.

What this changes in practice is large. A network trained only to remove noise has learned, without being asked, which way the data lie. A diffusion model is such a denoiser applied over and over, from pure noise down to an image. Interviewers ask for this link often, so be ready to say it in one sentence. The best denoiser is the noisy input plus $\sigma^2$ times the score.

The cell checks the whole example in three ways. It computes the denoiser matrix from the formula. It fits the best linear map on 50,000 noisy samples by least squares. And it checks Tweedie's formula by computing the score with finite differences.

The fitted map agrees with the formula to the second decimal, with a largest difference of 0.003. Its error matches $8/7$, and Tweedie's formula holds to the printed precision. For Gaussian data the best denoiser is linear, so a linear least-squares fit finds it. For any other data the conditional mean is nonlinear. The Wiener filter is then only the best linear denoiser, and a neural denoising autoencoder can beat it.

import numpy as np

def gaussian_denoiser(Sigma, sigma2):
    raise NotImplementedError

def score_gaussian(xt, Sigma, sigma2):
    raise NotImplementedError

Regularized autoencoders, and what each one penalizes

An overcomplete autoencoder will learn to copy unless something stops it. Even an undercomplete one, with a powerful encoder and decoder, can compress well without capturing any useful structure. Each regularized autoencoder adds a different pressure. Below, $\operatorname{Enc}_\phi$ is the encoder, $\operatorname{Dec}_\theta$ the decoder, and $\mathbf{z}$ the code.

Sparse. Make most code entries zero for any given input. One way is a penalty $\alpha_s\lVert\mathbf{z}\rVert_1$, the sum of the absolute values of the code entries, scaled by a weight $\alpha_s$. Another is a TopK activation, which keeps the $k$ largest entries and zeroes the rest. The code can then be wider than the input, because each input uses only a few of its entries. This is how sparse autoencoders split a language model's internal activations into many features, each active on few inputs and often interpretable. Gao et al. (2024) trained one with 16 million latents on GPT-4's activations.

Contractive. Penalize how much the code changes when the input changes (Rifai et al. 2011). The penalty is $\alpha_c\,\lVert\partial\operatorname{Enc}_\phi(\mathbf{x})/\partial\mathbf{x}\rVert_F^2$, the squared Frobenius norm of the encoder's Jacobian, the matrix of partial derivatives of each code entry with respect to each input entry. The reconstruction term wants the code to change along the data surface, where inputs really differ. The penalty wants it to change nowhere. The compromise is a code that is sensitive only along the surface. Alain and Bengio (2014) showed that a denoising autoencoder with small Gaussian noise acts like a contractive penalty on the whole map from input to reconstruction.

Masked. Hide a large random subset of the input and reconstruct it from the rest. This is denoising with a corruption that blanks out whole patches. The masked autoencoder (MAE) of He et al. (2022) cuts an image into square patches (§27.1) and hides 75 percent of them. It encodes only the visible quarter with a Vision Transformer and computes the loss on the hidden patches only. The goal there is the encoder's representation, not the reconstruction, and §28.5 covers it in depth.

Variational. Make the code random, and pull its distribution toward a fixed simple distribution called the prior. The pull is a penalty on the KL divergence between the two, a measure of how far one distribution is from another (§0.5). That is the variational autoencoder (VAE) of §32.3, and the rest of this chapter explains why it is the one that can generate.

Autoencoders at real scale

The autoencoder of latent diffusion is the one you are most likely to meet at work. Rombach et al. (2022) trained a family of convolutional autoencoders on the OpenImages dataset and evaluated them on ImageNet validation images. The one with $f = 8$ and $c = 4$ is the autoencoder of Stable Diffusion 1. The paper calls this kind of autoencoder KL-regularized, and its released checkpoint is named KL-f8, because a small KL term pulls its code toward a Gaussian (the idea of §32.3). Its training loss is not plain squared error. It adds a perceptual loss and an adversarial loss, which §32.6 develops. Squared error alone would reward exactly the blurry conditional means of the previous section.

Three numbers measure reconstruction quality. The peak signal-to-noise ratio (PSNR) is $10\log_{10}(\text{range}^2/\text{MSE})$ in decibels, where MSE is the mean squared pixel error and range is the span of pixel values. Higher is better. The structural similarity index (SSIM) compares local brightness, contrast, and structure, and equals 1 for identical images. The reconstruction FID compares the statistics of features that an image classifier extracts from real and from reconstructed images, so lower is better. It is defined properly with the other evaluation metrics for generators.

The same table separates how many numbers the latent keeps from how they are arranged. Every 6 dB of PSNR halves the typical pixel error, since $20\log_{10}2 \approx 6$. The $f = 4$, $c = 3$ model keeps three times as many numbers (1/16 of the image) and reaches 27.53 dB. The $f = 16$, $c = 16$ and $f = 32$, $c = 64$ models keep exactly as many numbers as KL-f8, 1/48 of the image, on coarser grids with more channels. They fall to 24.08 dB and 22.27 dB. Keeping fewer numbers costs more, so $f = 16$, $c = 8$ (1/96 of the image) reaches only 21.94 dB.

Later models chose to keep more. Stable Diffusion 3 widened the latent to 16 channels, because more channels reconstructed markedly better and a larger generator could make use of them (Esser et al. 2024). Generators do their work in this latent space. A diffusion Transformer (DiT) is a Transformer that denoises a grid of latent patches (Peebles and Xie 2023). It cuts the latent into small square patches, 2 by 2 in its best models, and treats each patch as one token.

Reconstruction error as an anomaly score

Reconstruction error has a second, very practical use. An autoencoder trained on normal data rebuilds normal inputs well and unusual ones badly, so its error is an anomaly score. Teams use it to monitor sensors, payments, and model activations. The linear version, a PCA reconstruction, is a strong baseline that is easy to explain to a reviewer.

The engineering is all in the threshold. Suppose you want a false positive rate of 1 percent, meaning 1 percent of normal inputs raise a false alarm (§0.11). Then the threshold is the 99th percentile of scores on normal data the model was not fit to. Training scores are biased low, because the model was fit to make exactly those errors small. Test data must never be used to set the threshold. And the rule must use only $\mathbf{x}$ and fixed, stored parameters, so it can score a single example on its own.

import numpy as np

def fit_anomaly_detector(X_train, X_val, d_z, fpr):
    raise NotImplementedError

def flag(det, X):
    raise NotImplementedError

Why a plain autoencoder cannot generate

We now have a network that turns images into short codes and codes back into images. To generate new images, it seems we only need to make up a new code and decode it. But which code? The training loss never said where codes should live. The encoder was free to put them anywhere, at any scale, in any shape, with any gaps. The decoder was trained only on the codes the encoder actually produced. Anywhere else, its output is an extrapolation that nothing constrained.

Go back to the four arcs. Their codes filled four intervals of the line, with empty stretches between them. The natural first attempt is to fit a Gaussian to the codes and sample from it. The cell compares that with two other sources of codes. It counts a decoded point as off the data when it is more than 0.1 from every training point, which is 2.5 times the radial noise.

The widget below shows an autoencoder of the same kind, trained separately, so its percentages differ from the cell's. On top is the code line, with the density that new codes are drawn from, the training codes, and the sampled codes. Below are the data, the dashed curve the decoder draws as the code sweeps along the line, and the decoded samples. Point at the code line to see where the decoder sends that code.

The reconstructions sit on the data, as they should. But in the cell a Gaussian fitted to the codes sends about a quarter of its samples (25.9 percent, 518 of 2,000) off the data, and in the widget more than a third. Of the cell's 518, 453 are codes in the gaps between the intervals. The decoder maps those codes onto bridges it draws between the arcs, and it cannot avoid drawing them. A continuous decoder maps the connected code line onto a connected curve, so a curve that visits four separate arcs must cross the gaps. The other 65 are codes beyond the two ends of the code range, where the decoder continues the curve past the end of an arc and off the circle. What a good generator can do is put very little probability on the codes that land there.

There is a catch. With a very narrow bandwidth, the density estimate only replays training codes, which is memorization, not generation. Widen it and the holes come back. A useful generator needs a code distribution that is simple to sample, matched to the codes, and smooth enough to generalize, all at once.

There are two ways to get one, and the rest of this part of the book uses both.

• Shape the codes during training. Make the code a random variable and choose a simple prior, such as $\mathcal{N}(\mathbf{0}, \mathbf{I})$, in advance. Then add a term to the loss that pulls the codes' distribution toward it. Sampling then means sampling the prior. This is the variational autoencoder, which §32.2 derives from a probabilistic model and §32.3 implements.
• Model the codes after training. Keep the autoencoder, and fit a powerful generative model to its codes. This is what Stable Diffusion does. Its autoencoder carries only a small KL term, which stops the variance of the latent values from growing without bound. The latents are then rescaled to unit standard deviation, and a diffusion model in the latent space does the generating. With discrete codes, the second model can be a Transformer trained on code indices, the route of §32.5.

A third idea drops reconstruction altogether. It trains a network to predict the representation of one part of an input from another part, in latent space, so it never spends capacity on unpredictable pixel detail. That is the idea behind joint-embedding predictive architectures, a different answer to the same question of which features deserve to survive.

Read more

• Baldi & Hornik (1989). Neural networks and principal component analysis: Learning from examples without local minima. The proof that a linear autoencoder's global minima are the principal subspace and that every other critical point is a saddle.
• Kunin, Bloom, Goeva & Seed (2019). Loss Landscapes of Regularized Linear Autoencoders. Weight decay makes the decoder's left singular vectors the principal directions, with the full critical-point analysis and the link to probabilistic PCA.
• Hinton & Salakhutdinov (2006). Reducing the Dimensionality of Data with Neural Networks. Deep nonlinear autoencoders beat PCA by a wide margin on digits, faces, and documents. Four pages.
• Vincent, Larochelle, Lajoie, Bengio & Manzagol (2010). Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. The denoising criterion, its manifold interpretation, and why it beats a plain bottleneck.
• Alain & Bengio (2014). What Regularized Auto-Encoders Learn from the Data Generating Distribution. The precise sense in which a denoising or contractive autoencoder estimates the score of the data.
• Rombach, Blattmann, Lorenz, Esser & Ommer (2022). High-Resolution Image Synthesis with Latent Diffusion Models. The autoencoder behind Stable Diffusion. Table 8 lists every downsampling factor and channel count with its reconstruction quality.
• He, Chen, Xie, Li, Dollár & Girshick (2022). Masked Autoencoders Are Scalable Vision Learners. Masking 75 percent of patches turns a denoising autoencoder into a strong self-supervised pretraining method.
• Goodfellow, Bengio & Courville (2016). Deep Learning, chapter 14, Autoencoders. A textbook treatment of undercomplete, sparse, denoising, and contractive autoencoders and their manifold view.

Watch next

▶

▶

▶