How the diffusion and flow models of the last chapter become systems that turn a sentence into a photograph or a minute of film. Latent spaces, diffusion transformers, text encoders, autoregressive and masked token generators, video at scale, honest evaluation, and control, with every cost counted and every published number checked.

Chapter 34 · Section 34.1 · 80 min

Latent diffusion

Latent diffusion is how nearly every large image and video generator works. It does not run a diffusion model on the pixels of an image. It first shrinks each image into a much smaller grid of numbers with an autoencoder, a pair of networks that compress an image and expand it back. The diffusion model learns to generate those small grids. The second half of the autoencoder turns a generated grid into a picture once, at the very end. Stable Diffusion, SDXL, Stable Diffusion 3, FLUX, and most video models are built this way.

The reason is cost. A diffusion model or a flow model starts from pure noise and calls a network dozens of times. Each call moves the sample a little closer to clean data. §33.6 built one that turns noise into points on a ring in the plane. A 512 by 512 color photograph has $512\cdot512\cdot3 = 786{,}432$ numbers, and every call reads and writes all of them.

Two real networks show what that costs. ADM, the strongest diffusion network on pixels in 2021 (Dhariwal and Nichol 2021), needs 1,983 billion multiply-adds per call at 512 by 512. DiT-XL/2, a transformer that denoises a compressed $64\times64\times4$ version of the same image, needs 524.6 billion, about a quarter. Its samples are also better. With guidance it reaches an FID of 3.04, against 3.85 for ADM with classifier guidance and a separate upsampling model, which costs more still (Peebles and Xie 2023, Section 5.1 and Table 4). FID is the standard distance between generated and real images, lower is better, and §34.6 derives it.

The saving rests on one observation (Rombach et al. 2022). Most of the numbers in a photograph describe detail you would not miss, such as the grain of a texture or the noise of a camera sensor. A model on pixels spends its capacity, its training, and every sampling step on that detail. You met the fix in §1.10, where Stable Diffusion turned a photograph into a $64\times64\times4$ grid, 48 times fewer numbers, before generating anything.

This section builds latent diffusion and counts what it buys and what it costs. We define the model, count the numbers, tokens, and FLOPs it saves, and prove its price, a floor on quality that no generator can get below. Then we fix the two things that go wrong when the data is a latent rather than an image, its scale and its size. We follow one prompt through a whole generation, compare Stable Diffusion 1.5, SDXL, SD3, and FLUX, and end with the alternatives and with decoding very large images. Most interview questions on this topic are arithmetic of exactly this kind, done on a whiteboard.

Pixels are mostly detail nobody sees

Compressing an image does two different jobs. Perceptual compression removes fine detail that a viewer would not miss. Semantic compression keeps what the image is about, its layout, its objects, and its light. Rombach et al. (2022) separate the two jobs. A diffusion model is an excellent but expensive tool for the second job and a wasteful one for the first. Deciding where a dog sits does not require modeling every hair of its fur.

So the work is split in two. An autoencoder does the perceptual compression. Its encoder maps an image to a smaller grid of vectors, called the latent. Its decoder maps the grid back to pixels and paints in plausible fine texture. The autoencoder is trained once and then frozen, never updated again. The diffusion model then does the semantic part inside the small grid. The split pays off because the two halves run at different rates. The autoencoder runs once per image, and the diffusion model runs dozens of times.

Here is the split in miniature, with an encoder so simple you can run it by hand.

✏️ Pooling an 8 by 8 image by hand

Let the encoder average each 2 by 2 block of an 8 by 8 grayscale image, and let the decoder copy each average back into its block. The image has 64 numbers and the latent has 16, a ratio of 4.

Look at one block, with entries $a, b$ in its top row and $c, d$ in its bottom row. The encoder keeps the mean $m = (a+b+c+d)/4$ . The decoder writes $m$ four times. The error at each entry is its distance from that mean. Two kinds of content behave very differently.

A smooth ramp, say $a, b, c, d = 1.00, 1.50, 1.25, 1.75$ , has mean $1.375$ and errors $-0.375, 0.125, -0.125, 0.375$ . The decoder keeps the overall level and loses only the slope inside the block.
A pixel checkerboard $+\delta, -\delta, -\delta, +\delta$ has mean exactly 0. Its latent is zero, so the decoder cannot bring back any of it. The squared error is $4\delta^2$ per block.

The cell below builds an image as a ramp plus a checkerboard of amplitude $\delta = 0.5$ . Over the 16 blocks, the checkerboard carries 16 units of squared error, and all of it is lost. The ramp loses 5 units. The two losses add because the two patterns are orthogonal.

Python · cell 1

The smooth part of the image survived the round trip and the pixel-level texture did not. A learned decoder does better than copying, since it can paint in a texture that looks right. But it cannot know which checkerboard was there. A real autoencoder makes the same trade as block averaging, only far better. It keeps the content and gives up exact detail.

The autoencoder inside Stable Diffusion keeps 4 learned numbers for every 8 by 8 block of RGB pixels. It is trained with the three losses that §32.6 built for image tokenizers.

An L1 pixel error, the mean absolute difference between the image and its reconstruction.
The LPIPS perceptual distance, a distance between the features that a pretrained image network extracts from the two images.
A patch discriminator, a small network trained to tell real image patches from reconstructed ones. Fooling it pushes the decoder toward realistic texture.

The encoder outputs a Gaussian for every latent number rather than a single number, as a variational autoencoder does (§32.3). A KL penalty pulls each Gaussian toward a standard normal. Its weight is about $10^{-6}$ , so small that the encoder is almost deterministic, and the latent behaves like an ordinary compressed code. In particular, the penalty does not pin down the latent's overall scale. That is why the definition below carries a scale factor.

The latent diffusion model, defined

Here is the whole system in one place. Write $\mathbf{x} \in \R^{H\times W\times 3}$ for an image with $H$ rows and $W$ columns of pixels, with values scaled to $[-1, 1]$ . Write $\mathbf{c}$ (bold) for whatever the model is conditioned on, such as a class label or a sentence. The italic $c$ is a different thing, the number of latent channels.

📐 Latent diffusion model

A latent diffusion model has two stages.

An autoencoder, trained first and then frozen. The encoder $\mathcal{E}: \R^{H\times W\times 3} \to \R^{h\times w\times c}$ has spatial downsampling factor $f$ , so $h = H/f$ and $w = W/f$ , and $c$ latent channels. The decoder $\mathcal{D}: \R^{h\times w\times c} \to \R^{H\times W\times 3}$ maps back. Suppose the encoder outputs a Gaussian with mean $\boldsymbol\mu(\mathbf{x})$ and standard deviation $\boldsymbol\sigma(\mathbf{x})$ . Then $\mathcal{E}(\mathbf{x})$ is either the sample $\boldsymbol\mu(\mathbf{x}) + \boldsymbol\sigma(\mathbf{x})\odot\boldsymbol\epsilon'$ with $\boldsymbol\epsilon' \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ (as in Rombach et al. 2022) or the mean $\boldsymbol\mu(\mathbf{x})$ (as in FLUX's reference code).
A normalization and a generator. Two constants, the latent scale $s_z > 0$ and the latent shift $m_z$ , are fixed from data statistics before training. They define the normalized clean latent

\mathbf{z}_0 = s_z\big(\mathcal{E}(\mathbf{x}) - m_z\big) \in \R^{h\times w\times c}.

A diffusion or flow model $p_\theta(\mathbf{z}_0\mid\mathbf{c})$ is trained on these latents with the losses of §33.2 or §33.6.

To sample, draw $\mathbf{z}_0 \sim p_\theta(\cdot\mid\mathbf{c})$ by running a sampler from pure noise, then decode $\hat{\mathbf{x}} = \mathcal{D}\big(\mathbf{z}_0/s_z + m_z\big)$ .

In plain words, compress every training image once, standardize the codes, learn to generate codes, and decode a generated code once at the end. Nothing in the diffusion machinery changes. Only the data does.

The training loss is the flow-matching loss of §33.6, applied to latents. Here it is again in brief. A clean latent at time $t = 0$ and a noise sample $\boldsymbol\epsilon$ at $t = 1$ are joined by a straight line, $\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\boldsymbol\epsilon$ . Moving along that line at constant speed has velocity $\boldsymbol\epsilon - \mathbf{z}_0$ . The network $\mathbf{v}_\theta$ sees the point $\mathbf{z}_t$ and the time $t$ and learns to predict that velocity. This path is called rectified flow. This book, SD3, and FLUX all put data at $t = 0$ and noise at $t = 1$ .

Training draws a time $t \in (0, 1)$ from a density $\pi(t)$ and a noise sample $\boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ of the latent's shape, and minimizes

\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x},\,\mathbf{c},\,t\sim\pi,\,\boldsymbol\epsilon}\,\big\lVert \mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) - (\boldsymbol\epsilon - \mathbf{z}_0)\big\rVert^2, \qquad \mathbf{z}_0 = s_z\big(\mathcal{E}(\mathbf{x}) - m_z\big).

In plain words, encode an image, normalize it, mix it with noise in a random proportion, and ask the network which way the straight line from data to noise points. SD3 draws $t$ from a logit-normal density, $t = 1/(1 + e^{-u})$ with $u$ standard normal, which practices the middle of the path most (§33.6 explains why). The original latent diffusion model used the noise-prediction loss of §33.2 instead, $\lVert\boldsymbol\epsilon_\theta(\mathbf{z}_t, t, \mathbf{c}) - \boldsymbol\epsilon\rVert^2$ with $\mathbf{z}_t = \alpha_t\mathbf{z}_0 + \sigma_t\boldsymbol\epsilon$ . In both, no gradient reaches $\mathcal{E}$ or $\mathcal{D}$ .

Freezing the autoencoder has a practical payoff that is easy to miss. Its output for a given image never changes, so large training runs encode the whole dataset once and train on the stored latents. SD3 does exactly this (Esser et al. 2024, Appendix E.1). The encoder then costs nothing per training step, and each step touches 48 times fewer numbers than a pixel model would at Stable Diffusion's settings.

Labs ask candidates to write this training step. The math is short, and every convention in it is a place to slip.

Interview

The latent flow-matching training step

You are given a frozen encoder frozen_encoder(x) and a velocity network velocity_model(z_t, t). The encoder maps images x of shape (B, H, W, 3) to a pair (mu, logvar), each of shape (B, h, w, c), channels last. The network takes latents of shape (B, h, w, c) and times of shape (B,).

Implement latent_fm_loss(x, t, eps_enc, eps, scale, shift) in four steps.

Sample the latent by reparameterization, $\boldsymbol\mu + \boldsymbol\sigma\odot$ eps_enc, where eps_enc is a given standard normal array.
Normalize it, $\mathbf{z}_0 = s_z(\mathcal{E}(\mathbf{x}) - m_z)$ , with scale $= s_z$ and shift $= m_z$ .
Form $\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\boldsymbol\epsilon$ with the given noise eps.
Return, as a Python float, the mean over every entry of the squared error between the network's output and the target $\boldsymbol\epsilon - \mathbf{z}_0$ .

import numpy as np

def latent_fm_loss(x, t, eps_enc, eps, scale, shift):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Counting what the latent saves

Two whole numbers describe the autoencoder, the factor $f$ and the channel count $c$ . A third, the patch size $p$ , describes how a transformer denoiser reads the latent. A transformer works on tokens, a sequence of vectors. To make tokens from a grid, it cuts the grid into $p\times p$ squares, called patches, and flattens each square into one vector. The Vision Transformer of §27.2 does the same to pixels.

Three real ratios, worked by hand. Stable Diffusion 1 uses $f = 8$ and $c = 4$ , so $\rho_{\text{ae}} = 3\cdot64/4 = 48$ . SD3 and FLUX.1 use $f = 8$ and $c = 16$ , so $\rho_{\text{ae}} = 12$ , a quarter of the compression, for reasons the quality floor below makes clear. The deep-compression autoencoders (DC-AE) of Chen et al. (2024) reach $3\cdot1024/32 = 96$ with $f = 32$ and $c = 32$ . Their models with $f = 64$ , $c = 128$ and $f = 128$ , $c = 512$ have the same ratio of 96, because they trade spatial resolution for channels.

How much compression is right? Rombach et al. (2022) trained the same class-conditional ImageNet model in latent spaces with $f \in \{1, 2, 4, 8, 16, 32\}$ , with the same compute, number of steps, and parameter count.

The other side of the choice is how well each autoencoder reconstructs. Two scores measure it. The reconstruction FID (rFID) is the FID between real images and their own reconstructions, so lower is better. The PSNR, peak signal-to-noise ratio, is $10\log_{10}$ of the squared pixel range over the mean squared error, in decibels, so higher is better. Each 3 dB more means about half the squared error. The latent diffusion paper reports both for autoencoders with the KL penalty, trained on OpenImages and tested on ImageNet validation images (its Table 8).

$f$	$c$	$\rho_{\text{ae}}$	rFID	PSNR (dB)
2	2	6	0.086	32.47
4	3	16	0.27	27.53
8	4	48	0.90	24.19
16	16	48	0.87	24.08
16	8	96	2.63	21.94
32	64	48	2.04	22.27
32	16	192	7.3	20.38

Reconstruction gets worse as the ratio grows. At the same ratio of 48, though, the three autoencoders still differ, so the ratio is not the whole story. More spatial downsampling at the same ratio means fewer, fatter latent vectors. That is good for the denoiser, which sees fewer tokens, and harder for the autoencoder to learn.

The denoiser's cost, step by step

Now the network that runs many times. A FLOP is one floating-point operation, and a multiply-add counts as two. A transformer has $L$ blocks (its depth), and each token is a vector of $d$ numbers (its width). §5.6 counted the forward FLOPs of one block exactly, and §27.2 applied the count to images. Here it is for a diffusion transformer.

∂ Counting the two terms

Per block, the attention projections $\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V, \mathbf{W}_O$ are four $d\times d$ matrices, and the MLP has a $d\times 4d$ and a $4d\times d$ matrix. That is $12d^2$ weights, each doing one multiply-add per token, so $24Td^2$ FLOPs. The scores $\mathbf{Q}\mathbf{K}^\top$ and the weighted sum $\mathbf{A}\mathbf{V}$ each take $T^2d$ multiply-adds summed over the heads, so $4T^2d$ FLOPs. Multiply both by $L$ . The ratio of the two terms is $4T^2d/(24Td^2) = T/(6d)$ , which exceeds 1 exactly when $T > 6d$ . Since $T = HW/(fp)^2$ , doubling both sides multiplies $T$ by 4, a term linear in $T$ by 4, and a term in $T^2$ by 16. ∎

A sampler with $S$ steps makes $S$ network calls. The samplers of §33.4 use from one to a few hundred. Guidance doubles that, and interviewers probe this point. Classifier-free guidance (§33.5) runs the network twice per step, once with the prompt and once with an empty prompt, giving velocities $\mathbf{v}_c$ and $\mathbf{v}_\varnothing$ . It then moves along $\mathbf{v}_\varnothing + \gamma(\mathbf{v}_c - \mathbf{v}_\varnothing)$ with a guidance scale $\gamma > 1$ , further in the direction the prompt adds. So $S$ steps cost $2S$ evaluations. Stacking the two inputs into one batch halves the number of sequential calls and keeps the GPU busier, but the arithmetic is the same. A guidance-distilled model such as FLUX.1 [dev] takes the guidance scale as an input and needs one evaluation per step.

The cell counts numbers, tokens, and FLOPs for five real configurations.

Python · cell 2

Two rows can be checked against published numbers. DiT-XL/2 at 256 by 256 costs 237 GFLOPs (billions of FLOPs), which is 118.4 billion multiply-adds, against the 118.6 that Peebles and Xie (2023) list. They call these numbers Gflops, but the match shows that they count multiply-adds. At 512 it costs 524.3 billion multiply-adds against their 524.6. The small gaps are the patch embedding and the output head, which the formula leaves out. The SD3 and FLUX rows use the published depths and widths, so they are estimates in the same sense. SD3's width is 64 times its depth, 2,432 for depth 38 (Esser et al. 2024, Section 4). FLUX.1 has width 3,072 and 19 plus 38 blocks. The DC-AE row shows the point of deep compression. A 1024 by 1024 image costs what a 512 by 512 one did, because both give 1,024 tokens. And attention stays under a quarter of each call even for SD3, so at 1024 by 1024 the linear layers still dominate.

✏️ SD3 at 1024 by 1024, counted

The image has $3\cdot1024^2 = 3{,}145{,}728$ numbers. The $f = 8$ , $c = 16$ autoencoder gives a $128\times128\times16$ latent of 262,144 numbers. Patches of $2\times2$ give $T = 64\cdot64 = 4{,}096$ image tokens of 64 numbers each.

The text arrives as 154 vectors of width 4,096. The first 77 come from two CLIP text encoders, trained to match captions with images (Chapter 30). Their outputs are joined side by side and padded with zeros to width 4,096. The next 77 come from T5-XXL, the encoder of a large text-only language model (Esser et al. 2024, Appendix B.2, and §34.3). Image and text tokens attend to each other jointly, so attention runs over $4{,}250$ tokens.

With depth 38 and width 2,432, one call costs $24\cdot38\cdot4250\cdot2432^2 = 2.29\times10^{13}$ FLOPs in the linear layers and $4\cdot38\cdot4250^2\cdot2432 = 6.7\times10^{12}$ in attention. That is about $3.0\times10^{13}$ in all, with $T/(6d) = 0.29$ . Fifty guided steps make 100 evaluations, about $3\times10^{15}$ FLOPs. One H100-class accelerator running at 40 percent of its $10^{15}$ dense 16-bit FLOPs per second would take roughly 7.5 seconds (an estimate).

Doubling the side to 2048 gives $T = 16{,}538$ , past the crossover $6d = 14{,}592$ . Each call grows 6.4 times, to $1.9\times10^{14}$ FLOPs.

The widget puts the whole budget on one screen. A preset sets the model's width and depth. You then set the resolution, the patch size $p$ , the autoencoder, the steps, and whether guidance is on. Each autoencoder button is a real one, so $f$ and $c$ change together. The widget shows the tokens, the denoiser's FLOPs split into linear layers and attention, the decoder's share, and a rough time on one H100.

In practice these counts decide the design. Before any training run, a team writes down $T$ for its target resolution and checks it against $6d$ . An interviewer will ask you to do the same on a whiteboard.

Warm-up

A latent budget

Implement latent_budget(H, W, f, c, p), which returns the tuple (h, w, T, n_latent, ratio).

h and w are the latent height and width.
T is the number of image tokens that a denoiser with patch size p sees.
n_latent is the number of latent numbers, $hwc$ .
ratio is the compression ratio $3f^2/c$ , as a float.

Raise ValueError if $H$ or $W$ is not divisible by $fp$ . Return Python integers for the counts, even when the inputs are NumPy integers.

def latent_budget(H, W, f, c, p):
    h, w = H // f, W // f
    return h, w, h * w, h * w * c, 3 * f * f / c     # placeholder: ignores p and never checks shapes

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The autoencoder sets a floor on quality

The decoder is the last thing every generated image passes through. If it cannot draw a face as fine as the one in a training photo, no generator upstream can produce that face either. Even a perfect generator of latents only produces the latents of real images, and decoding them gives the autoencoder's reconstructions, flaws included. This part makes that intuition exact. We need a way to measure how far apart two distributions of images are, and a way to say that the decoder does not stretch distances too much.

📐 Wasserstein-1 distance, push-forward, and Lipschitz maps

For distributions $p$ and $q$ on $\R^n$ , the Wasserstein-1 distance is

W_1(p, q) = \inf_\pi \mathbb{E}_{(\mathbf{a}, \mathbf{b})\sim\pi}\lVert\mathbf{a} - \mathbf{b}\rVert ,

the least expected distance over all couplings $\pi$ , joint distributions whose two marginals are $p$ and $q$ . For a map $g$ , the push-forward $g_\# p$ is the distribution of $g(\mathbf{a})$ when $\mathbf{a}\sim p$ . The map $g$ is $L_g$ -Lipschitz if $\lVert g(\mathbf{a}) - g(\mathbf{b})\rVert \le L_g\lVert\mathbf{a} - \mathbf{b}\rVert$ for all $\mathbf{a}, \mathbf{b}$ .

In plain words, pair every sample of $p$ with a sample of $q$ so that the average distance between partners is as small as possible, and $W_1$ is that smallest average. A push-forward is what you get by running every sample through the map. A Lipschitz map never stretches a distance by more than the factor $L_g$ . $W_1$ is the $W_2$ of §33.6 with distance in place of squared distance, and the sliced Wasserstein distance of §33.7 averages one-dimensional versions of it.

✏️ A floor you can compute by hand

Let images have two pixels, and let the data be $(1, -1)$ or $(-1, 1)$ with probability one half each, a two-pixel checkerboard. Let the decoder output only flat images $(m, m)$ , like the block-averaging decoder above. Its range is the diagonal line of the plane.

Each data point is at distance $\lvert 1 - (-1)\rvert/\sqrt2 = \sqrt2 \approx 1.41$ from the diagonal, at the point $(0, 0)$ . Any generator followed by this decoder outputs points on the diagonal, so every pairing with the data costs at least $\sqrt2$ on average. A generator that always outputs $m = 0$ reaches exactly $\sqrt2$ . So the best possible system is $\sqrt2$ from the data, and that is also the mean reconstruction error of the best encoder.

The theorem says the same thing for any data, decoder, and generator.

∴ The reconstruction floor

Let $p$ be the data distribution on images, $\mathcal{E}_\# p$ the distribution of encoded data, and $q$ any distribution on latents, the generator's. Let $\mathcal{D}$ be an $L_{\mathcal{D}}$ -Lipschitz decoder with range $\mathcal{R} = \{\mathcal{D}(\mathbf{z}) : \mathbf{z}\in\R^{h\times w\times c}\}$ , the set of all images it can output. Then

\mathbb{E}_{\mathbf{x}\sim p}\, \operatorname{dist}(\mathbf{x}, \mathcal{R}) \;\le\; W_1(\mathcal{D}_\# q,\; p) \;\le\; L_{\mathcal{D}}\, W_1(q,\; \mathcal{E}_\# p) + \mathbb{E}_{\mathbf{x}\sim p}\,\lVert\mathcal{D}(\mathcal{E}(\mathbf{x})) - \mathbf{x}\rVert ,

where $\operatorname{dist}(\mathbf{x}, \mathcal{R}) = \inf_{\mathbf{y}\in\mathcal{R}}\lVert\mathbf{x} - \mathbf{y}\rVert$ . If the encoder is the best possible for its decoder, so that $\lVert\mathcal{D}(\mathcal{E}(\mathbf{x})) - \mathbf{x}\rVert = \operatorname{dist}(\mathbf{x}, \mathcal{R})$ , the two bounds meet at $q = \mathcal{E}_\# p$ , and the floor is exactly the mean reconstruction error.

In plain words, the whole system can be no closer to the data than the decoder's range is, however good the generator. A perfect generator of latents gets you exactly as close as the autoencoder's reconstructions. Everything else the generator gets wrong is added on top, stretched by at most the decoder's Lipschitz constant.

∂ Proof of both bounds

Lower bound. Take any coupling $\pi$ of $\mathcal{D}_\# q$ and $p$ . Its first coordinate $\mathbf{y}$ always lies in $\mathcal{R}$ , so $\lVert\mathbf{y} - \mathbf{x}\rVert \ge \operatorname{dist}(\mathbf{x}, \mathcal{R})$ for every pair. Taking expectations gives $\mathbb{E}_\pi\lVert\mathbf{y} - \mathbf{x}\rVert \ge \mathbb{E}_p\operatorname{dist}(\mathbf{x}, \mathcal{R})$ , and the infimum over $\pi$ keeps the inequality.

Upper bound. $W_1$ is a metric, so the triangle inequality through the distribution of reconstructions $\mathcal{D}_\#\mathcal{E}_\# p$ gives

W_1(\mathcal{D}_\# q, p) \le W_1(\mathcal{D}_\# q, \mathcal{D}_\#\mathcal{E}_\# p) + W_1(\mathcal{D}_\#\mathcal{E}_\# p, p).

For the first term, let $\pi$ be any coupling of $q$ and $\mathcal{E}_\# p$ . Pushing both coordinates through $\mathcal{D}$ gives a coupling of $\mathcal{D}_\# q$ and $\mathcal{D}_\#\mathcal{E}_\# p$ with cost $\mathbb{E}_\pi\lVert\mathcal{D}(\mathbf{z}) - \mathcal{D}(\mathbf{z}')\rVert \le L_{\mathcal{D}}\,\mathbb{E}_\pi\lVert\mathbf{z} - \mathbf{z}'\rVert$ . Taking the infimum over $\pi$ gives $L_{\mathcal{D}}W_1(q, \mathcal{E}_\# p)$ . For the second term, the pair $(\mathcal{D}(\mathcal{E}(\mathbf{x})), \mathbf{x})$ with $\mathbf{x}\sim p$ is one coupling of the two distributions, with cost $\mathbb{E}_p\lVert\mathcal{D}(\mathcal{E}(\mathbf{x})) - \mathbf{x}\rVert$ , and $W_1$ is at most the cost of any coupling. When the encoder is optimal and $q = \mathcal{E}_\# p$ , the first term vanishes and the right side equals the left. ∎

Two honest caveats. FID is not $W_1$ , since it compares Gaussian fits to the features of an image classifier (§34.6). And a deep decoder's Lipschitz constant can be large. So the theorem explains a trend rather than giving a number to compute. The trend is real. Practitioners treat the autoencoder's rFID as a rough floor on the generation FID, and the SD3 authors write that the autoencoder's reconstruction quality upper-bounds the achievable image quality (Esser et al. 2024, Section 5.2.1). Fine text, small faces, and hands are often lost by the autoencoder before the generator ever runs, and no amount of denoiser training brings them back.

The theorem can be checked exactly on a linear autoencoder. Cut images into $f\times f$ patches of RGB pixels, and keep the top $c$ principal components of each patch, the linear autoencoder of §32.1. Its decoder has orthonormal rows, so it never stretches distances, and $L_{\mathcal{D}} = 1$ . Decoding after encoding is the orthogonal projection onto the decoder's range, so the encoder is optimal, and each reconstruction error is the distance to the range. The first cell measures reconstruction error against the compression ratio. Its random images have spectra that fall off with frequency like those of natural images.

Python · cell 3

At the same ratio of 48, the $f = 8$ , $c = 4$ coder (relative error 0.287) beats the $f = 4$ , $c = 1$ coder (0.334). For a linear coder this is guaranteed. Four $4\times4$ patches tile one $8\times8$ patch, so the $f = 4$ coder is one particular rank-4 linear map on $8\times8$ patches, and principal components give the best rank-4 map. Bigger patches let a coder exploit correlations across a wider area. For learned nonlinear autoencoders the catch is training, since autoencoders with large $f$ turned out to be hard to optimize. DC-AE solved that problem, as we will see below.

The second cell checks both bounds of the theorem on samples. It computes $W_1$ between two equal-size samples exactly, as the cheapest one-to-one pairing, which an assignment solver finds. The latent generator is the true latents plus Gaussian noise of relative size $\delta$ .

Python · cell 4

With a perfect latent generator ( $\delta = 0$ ), all three numbers agree at 6.028, the floor. As the generator gets worse, the system's distance grows, always between the two bounds. The inequalities hold exactly here, not just on average. The theorem applies to any pair of distributions, including the empirical distributions of two finite samples.

What this changes in practice is where you look first. If generated images lose fine text, compare against reconstructions of real images before touching the generator. An interviewer will ask why a better generator cannot fix a weak autoencoder, and this theorem is the answer.

Latents must be rescaled

A diffusion model's noise schedule mixes a clean sample and noise as $\mathbf{z}_t = \alpha_t\mathbf{z}_0 + \sigma_t\boldsymbol\epsilon$ , where $\alpha_t$ is the signal's weight and $\sigma_t$ the noise's weight at time $t$ (§33.1). Every schedule is designed for data whose coordinates have variance about 1. Pixels scaled to $[-1, 1]$ are roughly that size. Latents need not be. The KL penalty is too weak to pin their scale, and the raw latents of Stable Diffusion 1's autoencoder have a standard deviation of about 5.5.

Big latents look less noisy than the schedule intends. The measure of noisiness is the signal-to-noise ratio, $\operatorname{SNR}(t) = \alpha_t^2/\sigma_t^2$ , the variance of the signal part over the variance of the noise part for unit-variance data. Its natural log is the log-SNR $\lambda_t$ . One unit of log-SNR is one nat, a factor of $e$ in SNR. Here is a tiny case first. Take a time where the schedule intends an SNR of 1, with $\alpha_t = \sigma_t$ . A latent coordinate with standard deviation 5.49 has signal variance $5.49^2\alpha_t^2 \approx 30\,\alpha_t^2$ against noise variance $\sigma_t^2$ , so its true SNR is 30.

We need one more tool from §33.1. The time shift with parameter $\alpha$ maps each time $t$ to $\alpha t/(1 + (\alpha - 1)t)$ . Here $\alpha$ is a plain number, not the signal weight $\alpha_t$ . It keeps $t = 0$ and $t = 1$ fixed, and for $\alpha > 1$ it moves every time in between toward noise. On the straight path of rectified flow it lowers every log-SNR by exactly $2\log\alpha$ . SD3 introduced it to train at high resolution.

∴ Latent scale and effective SNR

Let $\mathbf{z}_0$ have per-coordinate variance $\sigma_z^2$ , and noise it as $\mathbf{z}_t = \alpha_t\mathbf{z}_0 + \sigma_t\boldsymbol\epsilon$ with $\boldsymbol\epsilon\sim\mathcal{N}(\mathbf{0}, \mathbf{I})$ independent of $\mathbf{z}_0$ . The per-coordinate signal-to-noise ratio is

\operatorname{SNR}_{\text{eff}}(t) = \frac{\operatorname{Var}(\alpha_t z_{0,i})}{\operatorname{Var}(\sigma_t\epsilon_i)} = \sigma_z^2\,\frac{\alpha_t^2}{\sigma_t^2} = \sigma_z^2\operatorname{SNR}(t), \qquad \lambda_{\text{eff}}(t) = \lambda_t + 2\log\sigma_z ,

so every log-SNR of the schedule is raised by the same constant $2\log\sigma_z$ . For the rectified-flow path, $\alpha_t = 1 - t$ and $\sigma_t = t$ , this is exactly the log-SNR of the unit-variance schedule after the time shift with $\alpha = 1/\sigma_z$ .

Proof. $\operatorname{Var}(\alpha_tz_{0,i}) = \alpha_t^2\sigma_z^2$ and $\operatorname{Var}(\sigma_t\epsilon_i) = \sigma_t^2$ , and the log of the ratio is $\log(\alpha_t^2/\sigma_t^2) + \log\sigma_z^2$ . For the last claim, the shift with parameter $\alpha$ lowers the log-SNR of the straight path by exactly $2\log\alpha$ , so $\alpha = 1/\sigma_z$ raises it by $2\log\sigma_z$ . ∎

In plain words, latents that are too big look less noisy than the schedule intends, at every step, by the same factor $\sigma_z^2$ in SNR. For SD1's latents, $\sigma_z \approx 5.49$ , so the offset is $2\log 5.49 = 3.41$ nats, a factor of 30. For rectified flow, oversized latents act like a time shift with $\alpha$ below 1, toward less noise.

Two different failures follow, depending on the schedule.

Variance preserving schedules, those with $\alpha_t^2 + \sigma_t^2 = 1$ , end at a small but nonzero SNR. Stable Diffusion's schedule ends at $\operatorname{SNR} = 0.0047$ ( $\alpha_t = 0.068$ at the last step), which is already the terminal-SNR problem of §33.1. With raw latents it ends at $0.14$ , a log-SNR of $-1.96$ instead of $-5.36$ . The image is plainly visible at the last training step. The noised latent no longer has unit variance either, about 1.14 at the end and about 30 at the start. Sampling starts from pure noise, which the network never saw in training.
Rectified flow ends at exactly zero SNR, so the end point is safe. But the whole curve moves as if the time shift had been applied with $\alpha = 1/\sigma_z$ , a shift toward less noise. The noisy times, where the layout of an image is decided, are squeezed into a sliver near $t = 1$ . That is the opposite of what a high resolution needs.

Rombach et al. (2022) met the first failure. On the raw latent the SNR was so high that the model committed to semantic detail early in the reverse process, and rescaling brought the SNR down (their Appendix D.1). Their rule, in Appendix G, measures one standard deviation $\hat\sigma$ over every entry of the first batch of latents, across all images, channels, and positions, and divides by it. For SD1's autoencoder that gives the well-known $s_z = 0.18215 = 1/5.49$ . The autoencoders in the same paper that round their latents to a codebook needed no rescaling, since their latents already have variance close to 1 (Appendix D.1).

The cell measures the offset on stand-in latents with standard deviation 5.49 under Stable Diffusion's schedule.

Python · cell 5

The measured offset is 3.41 at every step, as the theorem says. For rectified flow, the next cell checks the equivalence with a time shift. It uses the raw scale of the FLUX.1 autoencoder, $\sigma_z = 1/0.3611 = 2.77$ .

Python · cell 6

A model trained on raw FLUX latents at $t = 0.5$ is effectively trained at $t = 0.27$ , and at $t = 0.9$ it is effectively at $0.76$ . Most of its training would happen at low noise.

The widget shows both failures. The upper panel plots the log-SNR the schedule intends and the one raw latents actually get. The lower panel noises one row of latent values at six times, each strip drawn at its own scale so you see the balance of signal and noise.

Later models normalize with a shift as well as a scale, $\mathbf{z}_0 = s_z(\mathcal{E}(\mathbf{x}) - m_z)$ , both computed over a subset of the training data (Esser et al. 2024, Appendix B.2). The constants are part of each autoencoder's released configuration. Every pipeline must apply them in both directions, before training and before decoding.

📌 The latent constants of four autoencoders

autoencoder	$f$ , $c$	scale $s_z$	shift $m_z$	raw std $1/s_z$	offset $2\log(1/s_z)$ if forgotten
SD 1.x	8, 4	0.18215	0	5.49	$+3.41$
SDXL	8, 4	0.13025	0	7.68	$+4.08$
SD3	8, 16	1.5305	0.0609	0.65	$-0.85$
FLUX.1	8, 16	0.3611	0.1159	2.77	$+2.04$

The SD1 value is the default of the diffusers library's AutoencoderKL, and the SDXL value is in its released autoencoder configuration. The SD3 and FLUX.1 values are in diffusers' SD3 conversion script and in Black Forest Labs' reference code.

A per-channel normalization, with one mean and one standard deviation for each latent channel, is a reasonable refinement when channels have very different spreads. The exercise implements all three variants.

Core

Fit and apply a latent normalization

Latents arrive channels last, as an array Z of shape (N, h, w, c). Implement fit_latent_norm(Z, mode), which returns (shift, scale) for three modes.

"global" is the rule of the original latent diffusion paper. The shift is 0.0, and the scale is $1/\hat\sigma$ , with $\hat\sigma$ the standard deviation of all entries, computed around their mean.
"affine" is the rule of SD3 and FLUX. The shift is the mean of all entries, and the scale is one over their standard deviation.
"channel" gives one mean and one scale per channel, as arrays of shape (c,).

Every standard deviation here is the population one, which divides by the number of entries. Then implement normalize(Z, shift, scale), which returns $s_z(\mathbf{Z} - m_z)$ , and denormalize(z0, shift, scale), its exact inverse.

import numpy as np

def fit_latent_norm(Z, mode):
    return 0.0, 1.0 / np.var(Z)          # placeholder: a variance where a std belongs, one mode only

def normalize(Z, shift, scale):
    return scale * (Z - shift)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

⚠️ Five ways to get the latent scale wrong

Forgetting the scale entirely raises every log-SNR by $2\log\sigma_z$ . The model still trains, which is what makes the bug expensive.
Applying the scale twice, once in a data loader and once in the pipeline, lowers every log-SNR by the same amount.
Using one autoencoder's constants with another's latents, such as SD1's 0.18215 on SDXL latents, or forgetting FLUX's shift, leaves a leftover offset.
Estimating the statistics per image instead of over the dataset erases the brightness and color that each image's latent mean carries. The model can then no longer generate a dark image.
Decoding without inverting the normalization, $\mathcal{D}(\mathbf{z}_0)$ instead of $\mathcal{D}(\mathbf{z}_0/s_z + m_z)$ , gives washed-out or oversaturated images that look like a guidance problem.

When samples look wrong in a new pipeline, check these five before touching the model.

Bigger latents need more noise

A larger image needs more noise to hide its layout, and §33.1 showed why. Neighboring pixels repeat each other while the noise in them is independent. So averaging a block of $k^2$ pixels multiplies the SNR of the coarse structure by $k^2$ , and a schedule tuned at a small size leaves the layout readable at a large one. SD3 compensates with the time shift defined above, with $\alpha = \sqrt{m/n}$ for an image of $m$ pixels and a base resolution of $n$ pixels, which lowers every log-SNR by $2\log\alpha$ . Esser et al. (2024) derived the formula from a constant image. At 1024 by 1024 they then chose $\alpha = 3$ by a human preference study, where the formula from their 256 by 256 base says 4.

Two things change when the data is a latent. The first is harmless. With $f$ and $c$ fixed, the latent has $HW/f^2$ positions, so the ratio $m/n$ is the same whether you count pixels, latent positions, or tokens, and the formula needs no translation.

The second matters when the autoencoder itself changes. The constant-image argument counts independent noisy observations of the same signal, and in a latent every channel of every position is one such observation. So when $c$ or $f$ changes between the base model and the target, the count that matters is the number of latent numbers. Zheng et al. (2025) call it the effective data dimension, the number of tokens times their dimension,

N_{\text{eff}} = \frac{hw}{p^2}\cdot p^2c = hwc = \frac{HWc}{f^2}, \qquad \alpha = \sqrt{\frac{N_{\text{eff}}^{\text{target}}}{N_{\text{eff}}^{\text{base}}}} .

In plain words, count every number in the latent, not the tokens, and take the square root of the ratio. The patch size cancels, because cutting a grid into patches regroups its numbers without changing how many noisy observations of the image there are.

✏️ The shift for a representation autoencoder

Zheng et al. (2025) diffuse in the feature space of a frozen DINOv2-B encoder, a self-supervised vision transformer (§28.4), with a decoder trained to turn its features back into pixels. They call the pair a representation autoencoder. A 256 by 256 image becomes a $16\times16$ grid of 768-dimensional tokens, so $N_{\text{eff}} = 256\cdot768 = 196{,}608$ .

They take a base dimension of 4,096, citing Esser et al. For comparison, SD1's latent of a 256 by 256 image has $32\cdot32\cdot4 = 4{,}096$ numbers. The shift is $\alpha = \sqrt{196{,}608/4{,}096} = \sqrt{48} = 6.93$ , a log-SNR drop of $\log 48 = 3.87$ nats. Counting tokens alone would have said $\alpha = \sqrt{256/256} = 1$ , no shift at all. With the shift, their ImageNet generation FID went from 23.08 to 4.81 (their Table 4).

The count also runs the other way. Moving from SD1's $f = 8$ , $c = 4$ latent to DC-AE's $f = 32$ , $c = 32$ at the same resolution gives an $N_{\text{eff}}$ ratio of $(32/1024)/(4/64) = 1/2$ and $\alpha = 0.71$ , slightly less noise.

The constant-image argument assumes the observations are independent copies of one value. Latent channels are not. They are correlated, and a learned autoencoder may spread one feature over several of them. So treat $N_{\text{eff}}$ as the right first guess, and tune the final value as SD3 did. An interviewer who asks how you would set the noise schedule for a new autoencoder wants exactly this count, and the caveat.

Core

The shift for a new latent space

Describe a configuration by the tuple (H, W, f, c, p). Implement three functions.

effective_dim(cfg) returns the number of tokens times the numbers per token.
alpha_for(target, base) returns the shift $\alpha = \sqrt{N_{\text{eff}}^{\text{target}}/N_{\text{eff}}^{\text{base}}}$ .
shifted_times(t, target, base) applies $t\mapsto\alpha t/(1 + (\alpha - 1)t)$ elementwise to an array of times.

import numpy as np

def effective_dim(cfg):
    H, W, f, c, p = cfg
    return (H // (f * p)) * (W // (f * p))        # placeholder: counts tokens only

def alpha_for(target, base):
    return effective_dim(target) / effective_dim(base)

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Training in stages of resolution and aspect ratio

Every large image model trains in stages of increasing resolution. Most of what a model learns, such as objects, composition, and how text maps to content, is learned as well at 256 by 256 as at 1024, with a sixteenth of the tokens. Stable Diffusion 1 trained 237,000 steps at 256 before moving to 512 (its model card). SDXL pretrained 600,000 steps at 256 with batch size 2,048, then 200,000 at 512, then fine-tuned on several aspect ratios of about one megapixel each (Podell et al. 2023, Section 2.5). SD3 pretrained at 256, fine-tuned at 1024 with mixed aspect ratios, and switched on the time shift for the high-resolution stage.

Aspect ratios need care, because every image in a batch must have the same shape. SDXL sorts the data into buckets of different aspect ratios, each with close to $1024^2$ pixels and sides in multiples of 64. Every batch comes from one bucket, and the model is told the target size as a conditioning input (Podell et al. 2023, Section 2.3; §34.3 covers size and crop conditioning). Sides in multiples of 64 keep every latent side a multiple of 8, so the network's own downsampling stages divide evenly.

SD3 builds its grid of 2D positions for the largest height and width it will see, and crops from it for each bucket (Esser et al. 2024, Section 5.3.2). A transformer can instead pack images of different shapes into one sequence, with a mask that lets each image's tokens attend only to each other (Dehghani et al. 2023). That is the packing of §31.5.

One generation, end to end

Every Stable Diffusion-style pipeline has the same five parts.

A text encoder turns the prompt into a sequence of vectors, once. It runs once more for the empty prompt that guidance needs.
A latent of pure noise is drawn, in the latent's shape.
The denoiser runs $S$ steps. With classifier-free guidance each step takes two evaluations, and each step moves the latent along the guided velocity.
The finished latent is un-normalized, $\mathbf{z}_0/s_z + m_z$ .
The decoder turns it into pixels, once.

The image encoder is not used at all to generate from text. It is needed only for training and for editing a given image.

The two cells below run all five parts on a toy problem small enough to read every number. The images are 32 by 32 and show a red or blue square on the left or right, with four captions. The autoencoder is block averaging, as at the start of the section, with $f = 8$ , so each image becomes a $4\times4\times3$ latent of 48 numbers. The text encoder is a table with one random vector per word. The toy denoiser cannot read vectors, so the cell picks the training caption whose mean word vector is closest to the prompt's and conditions on it.

The denoiser is not a trained network. It is the exact best velocity for the stored training latents, the one a perfect network would learn, $\mathbb{E}[\boldsymbol\epsilon - \mathbf{z}_0\mid\mathbf{z}_t]$ , computed as in §33.6. It weighs every stored latent by how likely it is to have produced the noisy point, and those weights are a softmax, the soft lookup of §4.1.

Python · cell 7

Python · cell 8

Follow the numbers. The prompt becomes 6 tokens and a $6\times8$ context. The image's 3,072 numbers become a latent of 48, scaled by $s_z = 4.42$ . At $t = 1$ the model knows nothing and gives each caption a belief of 0.25. Its guided guess of the clean latent is far off, 151 percent, because a guidance scale of 3 overshoots by extrapolating three times the difference between the two guesses. One step later, at $t = 0.875$ , the model is 94 percent sure, and its guess is within 20 percent of the final latent. From then on it only refines details. Eight steps cost 16 evaluations and one decode, and the image is a blue square on the left, as asked.

The layout is decided at the noisiest steps. That is why the time shifts above move training toward high noise, where the decisions are made.

The widget runs the same toy pipeline in your browser, with its own random draws, so its numbers differ a little from the cell's. Press Play to walk through the five parts in turn, or click a part to jump to it. During the steps, the bars show how sure the unguided model is of each caption.

A real system replaces each toy part with a large network. The text tokens enter the denoiser through cross-attention or joint attention, whose maps show which image regions each word steers (§34.3). And the denoiser must generalize. The exact velocity of a finite training set can only ever return one of its stored latents, which is memorization (§34.6 measures it in real models).

✏️ One Stable Diffusion 1.5 generation, counted

The prompt goes through the text half of CLIP ViT-L/14, giving a $77\times768$ sequence of token vectors. A $64\times64\times4$ latent of noise has 16,384 numbers, 48 times fewer than the $786{,}432$ of the image. It enters an 860M-parameter U-Net (Podell et al. 2023, Table 1). A U-Net is a convolutional network that shrinks its feature maps and then grows them back, with shortcuts between maps of the same size (§34.2). This one cross-attends to the prompt's vectors at several resolutions.

One U-Net call costs 339 billion multiply-adds at this size, as measured on SD 1.4, whose U-Net is the same (Kim et al. 2024, Table 2). Fifty sampler steps with guidance make 100 evaluations, $3.4\times10^{13}$ multiply-adds, in 100 calls or, batched, in 50.

One decoder pass, counted layer by layer from the autoencoder's released configuration, costs about $1.26\times10^{12}$ multiply-adds, the price of 3.7 U-Net calls. That agrees with the 1.25 trillion that Kim et al.'s whole-pipeline total leaves for the decoder and text encoder together. So the decoder is 3.6 percent of this generation. Distill the sampler to 4 steps without guidance, $1.36\times10^{12}$ in the U-Net, and the same decoder becomes 48 percent of the cost.

Training the model took about 150,000 A100 hours, or 6,250 GPU days (the model card). In the later fine-tuning stages, ten percent of captions were dropped, so that one network also learned the unconditional prediction that guidance needs.

The other systems differ in what each part is, and the arithmetic above tells you what each change costs.

system	year	denoiser	latent	text encoder	sampler
SD 1.5	2022	860M U-Net	$f = 8$ , $c = 4$	CLIP ViT-L text half, about 123M	noise prediction, typically 50 guided steps
SDXL	2023	2.6B U-Net	$f = 8$ , $c = 4$	two CLIP text encoders, 817M	noise prediction, plus an optional refiner
SD3	2024	up to 8B joint-attention transformer	$f = 8$ , $c = 16$	two CLIP encoders and T5-XXL	rectified flow
FLUX.1	2024	12B transformer	$f = 8$ , $c = 16$	CLIP and T5-XXL	rectified flow, guidance-distilled [dev]
FLUX.2	2025	32B open-weight [dev]	$f = 8$ , $c = 32$ , a new autoencoder	a 24B vision-language model	rectified flow

The pattern is visible in the table. The denoiser grew from 0.9B to 32B parameters. The autoencoder stayed at $f = 8$ and grew from 4 channels to 16, then 32. The text encoder grew from CLIP's text half to a 24B vision-language model, and §34.3 explains why.

🔬 What each system changed

SDXL (Podell et al. 2023) generates $128\times128\times4$ latents at 1024 by 1024. Its U-Net moves the attention blocks to the lower resolutions, where attention is cheap, and its two text encoders are CLIP ViT-L and OpenCLIP ViT-bigG. The optional refiner is a second latent model specialized on the 200 lowest noise levels. It adds a little noise to the base model's latent and denoises it again, the noise-and-denoise editing of §34.7 (Meng et al. 2021).
SD3 (Esser et al. 2024) lets image and text tokens attend to each other in one attention, the design that §34.2 builds, and was counted above.
PixArt-α (Chen et al. 2023) shows the training side of the ledger. A 0.6B diffusion transformer on SD1's 4-channel latents, trained in careful stages on recaptioned data, took about 753 A100 GPU days in the latest version of the paper. That is 12 percent of Stable Diffusion 1.5's 6,250. The abstract on its arXiv page still gives an earlier estimate, 675.
FLUX.1 (Black Forest Labs 2024) is a 12B rectified-flow transformer on a 16-channel $f = 8$ latent with scale 0.3611 and shift 0.1159. Its [dev] model is guidance-distilled, so it needs one evaluation per step.
FLUX.2 (Black Forest Labs, November 2025) has a 32B open-weight [dev] model and uses the 24B-parameter Mistral-3 vision-language model as its text encoder. Its new autoencoder has 32 channels at $f = 8$ (its reference code). It was retrained from scratch for what its authors call a better trade-off between learnability, quality, and compression rate, where learnability means how easily a generator learns its latents.

Here is the whole pipeline in PyTorch, small enough to train on a laptop CPU. A frozen linear autoencoder ( $f = 4$ , $c = 4$ , principal components of patches) compresses 32 by 32 images of horizontal or vertical stripes. The latents are encoded once and normalized with one global shift and scale. A small convolutional velocity network is trained with logit-normal times and 10 percent condition dropout. Sampling uses Euler steps with batched guidance, then one decode.

🔥 In PyTorchfor a machine with PyTorch

With a production library the same steps are a few lines, and the normalization is the line people get wrong. In the diffusers library the autoencoder's constants live in its configuration.

Where the downsampling should happen

A diffusion transformer can reduce its tokens in two places. The autoencoder can downsample more, with a larger $f$ , or the transformer can use larger patches, with a larger $p$ . Both give $T = HW/(fp)^2$ . DiT's own sweep (Peebles and Xie 2023) showed that larger patches, and so fewer tokens, cost quality. Chen et al. (2024) asked the cleaner question of where to downsample at a fixed token count.

The obstacle was the autoencoder. Making Stable Diffusion's autoencoder downsample further in the standard way degraded its rFID on ImageNet 256 by 256 from 0.90 at $f = 8$ to 28.3 at $f = 64$ . DC-AE changed how such autoencoders are built and trained, and reached $f = 32$ to $128$ with good reconstruction. On ImageNet 512 by 512, the test was the U-ViT-H diffusion transformer (Bao et al. 2022). Against the standard $f = 8$ autoencoder, the $f = 64$ DC-AE made it 19.1 times faster at inference and 17.9 times faster to train, while FID with guidance improved from 3.55 to 3.01 (Chen et al. 2024, Table 3). Without guidance it was slightly worse, 13.96 against 11.04.

The arithmetic of this section predicts most of that speedup. The token count falls from $32\times32 = 1{,}024$ , a $64\times64$ latent cut into $2\times2$ patches, to $8\times8 = 64$ , 16 times fewer. The attention term falls even faster, 256 times, since it scales with $T^2$ . That is how the speedup can exceed 16.

Why not add channels and keep $f = 8$ ? Because reconstruction and generation pull in different directions, the trade-off that §32.6 measured. In SD3, 16 channels reconstructed far better than 4 or 8, with rFID 1.06 against 2.41 and 1.56. But a generator trained on them needed more capacity. Its gap to 8 channels shrank as the model grew and became negligible at depth 22 (Esser et al. 2024, Figure 10). A richer latent holds more fine detail, which is close to noise and hard to predict.

Cascades, latents, or pixels

Latent diffusion is not the only way to make large images. Two other designs compete with it.

A cascade generates a small image and then enlarges it with further diffusion models, each conditioned on the output of the last. Imagen (Saharia et al. 2022) runs a 2B-parameter model at 64 by 64, then super-resolution models of 600M and 400M parameters up to 256 and 1024. The trap is that errors pile up. Each upsampler is trained on real low-resolution images and then fed generated ones, with their small defects. The cascaded diffusion models (CDM) of Ho et al. (2022) fixed this with conditioning augmentation. They corrupt the low-resolution input with Gaussian noise or blur during training, so the upsampler learns to ignore small defects. Imagen's version is called noise conditioning augmentation. Their cascade reached ImageNet FIDs of 1.48 at 64, 3.52 at 128, and 4.88 at 256, with no classifier help.

A pixel-space model runs one diffusion model on the full image. Simple diffusion (Hoogeboom et al. 2023) gets there with a noise schedule shifted for resolution, a first layer that downsamples, and scaling up only the low-resolution part of the U-Net. Hourglass diffusion transformers (HDiT, Crowson et al. 2024) use a hierarchical transformer whose cost grows linearly with the number of pixels, and train directly at 1024 by 1024. Plain transformers with large pixel patches, trained to predict clean images, compete at 256 and 512 (JiT, Li and He 2025, recapped in §32.6).

	Latent diffusion	Cascade	Pixel space
What the main model sees	a frozen autoencoder's latent, 12 to 96 times fewer numbers	a small image, then upsampling stages	every pixel
Quality ceiling	the autoencoder's reconstructions (the floor theorem)	the upsamplers, with errors compounding across stages	none from compression
Cost per step at 1024 by 1024	lowest, and the decoder runs once	low at the base, high in the last stage	highest unless the architecture downsamples internally
Stages to train	two (autoencoder, then generator)	one per resolution	one
Editing and reuse	latents are shared by many models and tools	each stage separately	direct
Examples	SD 1 to 3, SDXL, FLUX, PixArt, most video models	Imagen, DALL-E 2's decoder, CDM	ADM, simple diffusion, HDiT, JiT

Practitioners largely chose latents, for the low cost per step, the single model, and the shared tools. The open question in 2026 is whether latents aligned with semantic features, or pixel transformers, will narrow the gap between reconstruction and generation. Whether to compress at all is settled. The same two-stage pattern returns twice more in the book. Video models compress time as well as space with causal 3D autoencoders (§34.5). Speech and music generators run on the tokens of neural audio codecs, which play the autoencoder's role for sound.

Decoding large images in tiles

The decoder runs once, but it runs at full resolution, and memory becomes the limit. The last stage of SD's decoder holds 128 channels at the output resolution. In 16-bit floats that is 268 MB per activation at 1024 by 1024, and 4.3 GB at 4096 by 4096. Its middle block has a self-attention layer over the latent positions. Stored whole, its score matrix would take 0.54 GB at 1024 by 1024 and 137 GB at 4096 by 4096. So large images are decoded in tiles. The naive way leaves visible seams. Each tile's convolutions see zero padding at the tile's border, where the full decoder would have seen the neighboring latents.

The fix rests on one fact about locality. Suppose the decoder's output at a latent position depends only on latents within a radius $r$ , its receptive field.

A real decoder is not strictly local, since its attention layer and its group-norm statistics see the whole tile. So production tiling is approximate, and the blending ramps hide what is left.

Interview

Tiled decoding without seams

The setup defines a local decoder decode(z) and a helper tile_starts(n, tile, ov). The decoder maps a latent tile of shape (a, b) to an output of shape (F a, F b) with F = 2. Its output at a latent position depends only on latents within radius R = 3, and it zero-pads at the edges of whatever it is given. The helper returns the start offsets of tiles of side tile, overlapping by at least ov, that cover range(n).

Implement blend_weights(n, starts, tile, r, ov). It returns an array of shape (len(starts), n) whose row $k$ is the weight of tile $k$ at each latent position. The weight is zero outside the tile. A cut edge is a tile edge that is not the start or end of the whole range. Near a cut edge the weight is zero for the first r positions, then rises linearly and reaches 1 after ov - 2r more positions. Precisely, with $m = \max(1, \text{ov} - 2r)$ , start $s_k$ , and end $e_k$ , the raw weight at position $u$ inside the tile is $\min(1, (u - s_k - r + 1)/m)$ for a cut on the left and $\min(1, (e_k - r - u)/m)$ for a cut on the right. Clip it below at 0, and take the product of the two when both edges are cuts. Finally, normalize each column to sum to 1.
Implement tiled_decode(z, tile, ov), which decodes a latent z of shape (h, w) tile by tile, never passing anything larger than tile by tile to decode. Blend the tile outputs with the outer product of the row and column weights, each repeated F times to reach output resolution. With ov at least 2 R, the result must equal decode(z) exactly.

import numpy as np

def blend_weights(n, starts, tile, r, ov):
    raise NotImplementedError

def tiled_decode(z, tile, ov):
    h, w = z.shape
    out = np.zeros((F * h, F * w))

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

🧭 What to remember

Latent diffusion trains the generator on the latents of a frozen autoencoder. Stable Diffusion's $f = 8$ , $c = 4$ autoencoder gives 48 times fewer numbers, $\rho_{\text{ae}} = 3f^2/c$ , and a transformer denoiser with patch size $p$ sees $T = HW/(fp)^2$ tokens.
One transformer call costs $24LTd^2 + 4LT^2d$ FLOPs. Attention dominates past $T = 6d$ , and doubling the resolution multiplies attention by 16. Guidance doubles the evaluations, and batching it saves calls, not FLOPs. In few-step models the decoder's single pass becomes a large share of the cost.
The reconstruction floor is a theorem. The system's $W_1$ distance to the data is at least the data's mean distance to the decoder's range. It is at most the latent generator's error, times the decoder's Lipschitz constant, plus the reconstruction error. No generator beats its autoencoder.
Latents must be normalized, $\mathbf{z}_0 = s_z(\mathcal{E}(\mathbf{x}) - m_z)$ , and the normalization inverted before decoding. Unscaled latents raise every log-SNR by $2\log\sigma_z$ , 3.41 nats for SD1. That breaks a variance preserving schedule's last step and, for rectified flow, acts as an unwanted shift toward less noise.
Bigger latents need more noise, and the count that matters is the number of latent numbers $hwc$ , not pixels or tokens. It gives SD3's formula value of $\alpha = 4$ (3 was chosen by preference) and the $\sqrt{48}$ of a DINOv2 latent.
A generation is five parts, a text encoder, a noise latent, $S$ guided denoising steps, un-normalization, and one decode, and the layout is settled in the first, noisiest steps. Deep-compression autoencoders, cascades, and pixel transformers are the alternatives, and large images are decoded in overlapping tiles blended with weights that vanish near cuts and sum to one.
We now know where to diffuse. The next question is what network should do it, the U-Nets, diffusion transformers, and joint-attention transformers of §34.2.

Rombach, Blattmann, Lorenz, Esser & Ommer (2022). High-Resolution Image Synthesis with Latent Diffusion Models. The paper behind Stable Diffusion. Read Section 4.1 for the downsampling sweep and Appendices D.1 and G for the latent scale.
Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. SD3. Section 5.2.1 on latent channels, 5.3.2 on the resolution shift and aspect ratios, and Appendix B.2 on the latent and text representations.
Podell et al. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. Resolution staging, aspect-ratio buckets, the refiner, and the Table 1 that compares Stable Diffusion's U-Nets.
Chen et al. (2024). Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. Autoencoders at $f = 32$ to 128, and the clean ablation of autoencoder downsampling against patch size.
Zheng, Ma, Tong & Xie (2025). Diffusion Transformers with Representation Autoencoders. A frozen representation encoder as the latent space, the width argument, and the dimension-dependent shift.
Ho et al. (2022). Cascaded Diffusion Models for High Fidelity Image Generation. The cascade alternative and noise conditioning augmentation.
Hoogeboom, Heek & Salimans (2023). Simple diffusion: End-to-end diffusion for high resolution images. How far pixel-space diffusion goes with a shifted schedule and a lighter high-resolution path.
Crowson et al. (2024). Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers. A hierarchical transformer whose cost grows linearly with pixels, trained directly at 1024 by 1024.

Made for larger screens

Latent diffusion

Pixels are mostly detail nobody sees

The latent diffusion model, defined

The latent flow-matching training step

Counting what the latent saves

The denoiser's cost, step by step

A latent budget

The autoencoder sets a floor on quality

Latents must be rescaled

Fit and apply a latent normalization

Bigger latents need more noise

The shift for a new latent space

Training in stages of resolution and aspect ratio

One generation, end to end

Where the downsampling should happen

Cascades, latents, or pixels

Decoding large images in tiles

Tiled decoding without seams

Read more

Watch next