How a network that only ever learns to remove a little noise can create images, video, audio, molecules, and even text. One connected argument from the forward noising process to DDPM, score matching, SDEs, fast samplers, guidance, and flow matching, with every step derived and every claim checked.

Chapter 33 · Section 33.1 · 60 min

Creating by denoising

A diffusion model makes new data, such as an image, by starting from pure noise and removing a little of the noise at a time. Each removal is a small, easy denoising problem, and a thousand of them in a row turn random static into a picture. To learn the removal, we first need the opposite process, the one that adds the noise. This section builds that noising process. We choose it in advance and never learn it, and everything a diffusion model learns, along with every sampler later in the chapter, is defined in terms of it.

This chapter matters because diffusion, together with its close relative flow matching (§33.6), is how most of today's image and video generators work, from Stable Diffusion and FLUX to the video models of §34.5. Most of them run inside the compact latent space of an autoencoder like those of §32.6, and Chapter 34 scales them up. This chapter builds the method itself, on toy data where every quantity can be checked exactly.

Other generators make an image in one network call. A variational autoencoder, or VAE, passes one draw of Gaussian noise through a decoder network and reads off an image (§32.3). Its samples blur, and §32.4 showed why. The decoder is trained with squared error, and the best answer under squared error is the average of all the images that fit the code (§32.1). An average of sharp images is a blurry image.

A generative adversarial network, or GAN, trains its generator against a second network that tries to tell generated images from real ones, the game that §32.6 uses as a loss. Its samples are sharp, but the game is delicate to train, and GANs cover less of the variety in the data than models trained by likelihood (Dhariwal & Nichol 2021). Both ask one network call to do all the work.

Diffusion splits the work into many small pieces. Picture eight small clusters of points arranged on a ring in the plane. Add a little Gaussian noise to every point, then a little more, a thousand times. The clusters swell, merge, and end as one round blob that has forgotten the ring. Now play the movie backward. Each frame differs from the one before it by only a little noise, so each backward step only has to remove a little noise. A network that learns that one small step can apply it a thousand times and turn the blob back into a ring.

This section is about the forward half of the movie, the noising. We define the noising process and find where a point lands after any number of steps. We show that a whole noise schedule is described by one curve, the signal-to-noise ratio. Then we compare the schedules that real systems use, find what breaks when the noise never quite reaches pure noise and when images get larger, and fix one notation for the whole chapter. The next section, §33.2, learns to run the movie backward.

The widget runs the movie forward on the ring. Press Play, or drag the time $t$ from 0, the clean data, to 1000, pure noise. The dashed circle has radius $\sqrt2$ , the root-mean-square distance of the points from the center. It is the same at every $t$ , because the process keeps the variance of each coordinate at 1, so the clean ring sits on it and pure noise fills it. The buttons choose the schedule, a recipe for how much noise to add at each step, and the four schedules are defined later in the section. On the right, the top panel shows how much of each original point survives and how large the noise is. The bottom panel shows the log of the ratio of their squares, with the steps where it is below $-5$ shaded.

Why many small steps are easier than one big one

Start with the extreme case, one step from data straight to heavy noise and one step back. Let the data be single numbers, $+1$ or $-1$ with equal probability. We corrupt a data point $x_0$ as $x_t = \alpha x_0 + \sigma\boldsymbol\epsilon$ . Here $\alpha$ is the share of the signal that is kept, $\sigma$ is the size of the noise, and $\boldsymbol\epsilon \sim \mathcal{N}(0, 1)$ is a standard Gaussian draw. A network trained with squared error to undo this in one shot learns the conditional mean $\mathbb{E}[x_0 \mid x_t]$ . That is the average of the clean values that could have produced $x_t$ , each weighted by how likely it is to have done so. §32.1 proved that it is the best possible guess under squared error.

✏️ One-shot denoising returns the average

By Bayes rule (§0.4), the chance that $x_0 = +1$ given $x_t$ is proportional to $\exp\!\big(-(x_t - \alpha)^2/(2\sigma^2)\big)$ , and the chance of $-1$ to $\exp\!\big(-(x_t + \alpha)^2/(2\sigma^2)\big)$ . Their ratio is $e^{2a}$ with $a = \alpha x_t/\sigma^2$ . So the conditional mean is $(e^{a} - e^{-a})/(e^{a} + e^{-a}) = \tanh(\alpha x_t/\sigma^2)$ .

Take heavy noise, $\alpha = 0.1$ and $\sigma^2 = 0.99$ . A noisy value $x_t = 1$ gives $\tanh(0.101) = 0.10$ , and $x_t = 2$ gives $0.20$ . For nearly every noisy value the best guess stays within about 0.25 of 0, the average of the data. It is neither $+1$ nor $-1$ .

Take light noise instead, $\alpha = 0.9$ and $\sigma^2 = 0.19$ . Now $x_t = 1$ gives $\tanh(4.74) = 0.9998$ , and the guess is nearly exact.

Under heavy noise, the honest answer to "where did this point come from?" is two spikes, at $+1$ and $-1$ , with nearly equal weight. That distribution, written $q(x_0 \mid x_t)$ , is the reverse of the corruption. No single number summarizes two spikes, so the squared-error answer splits the difference. This is the blur of a VAE in its purest form.

Now take a small step instead. Go from time $t-1$ to time $t$ by shrinking the point slightly and adding a little noise,

\mathbf{x}_t = \sqrt{1-\beta_t}\,\mathbf{x}_{t-1} + \sqrt{\beta_t}\,\boldsymbol\epsilon, \qquad \boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).

Here $\beta_t$ is a small variance, such as 0.001, and the identity covariance $\mathbf{I}$ means that every coordinate gets its own independent noise. Write $p_{t-1}$ for the density of $\mathbf{x}_{t-1}$ , the distribution of all the noisy points one step earlier. The reverse of this small step is nearly Gaussian.

In plain words, to undo a small step, undo the shrink, nudge the point slightly uphill on the log density of the previous frame, and add a little Gaussian noise. The uphill direction $\nabla\log p$ , the direction in which the density of noisy points rises fastest, is called the score. It runs the chapter from §33.3 on. That a diffusion with small steps can be reversed by steps of the same Gaussian form is a classical result about diffusion processes (Feller 1949). It is the starting point of diffusion models (Sohl-Dickstein et al. 2015).

∂ Why the reverse of a small step is nearly Gaussian

By Bayes rule,

q(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \frac{q(\mathbf{x}_t \mid \mathbf{x}_{t-1})\,p_{t-1}(\mathbf{x}_{t-1})}{p_t(\mathbf{x}_t)} .

The denominator does not depend on $\mathbf{x}_{t-1}$ , so look at the numerator as a function of $\mathbf{x}_{t-1}$ . The first factor is $\exp\!\big(-\lVert\mathbf{x}_t - \sqrt{1-\beta_t}\,\mathbf{x}_{t-1}\rVert^2/(2\beta_t)\big)$ up to a constant. Pull $\sqrt{1-\beta_t}$ out of the norm to get $\exp\!\big(-\lVert\mathbf{x}_{t-1} - \mathbf{m}\rVert^2/(2c_t)\big)$ , with $\mathbf{m} = \mathbf{x}_t/\sqrt{1-\beta_t}$ and $c_t = \beta_t/(1-\beta_t)$ . This is a narrow Gaussian bump of width about $\sqrt{\beta_t}$ .

Inside that bump a smooth $\log p_{t-1}$ is nearly linear, $\log p_{t-1}(\mathbf{x}_{t-1}) \approx \log p_{t-1}(\mathbf{m}) + \mathbf{g}^\top(\mathbf{x}_{t-1} - \mathbf{m})$ with $\mathbf{g} = \nabla\log p_{t-1}(\mathbf{m})$ . Add the two exponents and complete the square,

-\frac{\lVert\mathbf{x}_{t-1} - \mathbf{m}\rVert^2}{2c_t} + \mathbf{g}^\top(\mathbf{x}_{t-1} - \mathbf{m}) = -\frac{\lVert\mathbf{x}_{t-1} - \mathbf{m} - c_t\mathbf{g}\rVert^2}{2c_t} + \frac{c_t\lVert\mathbf{g}\rVert^2}{2}.

The last term does not depend on $\mathbf{x}_{t-1}$ , so it goes into the normalizing constant. What remains is the Gaussian $\mathcal{N}(\mathbf{m} + c_t\mathbf{g}, c_t\mathbf{I})$ . The dropped second-order term of $\log p_{t-1}$ changes this by a relative amount of order $c_t$ times the curvature, which vanishes as $\beta_t \to 0$ .

How small is small enough? On the $\pm1$ data we can measure it exactly. Every distribution in sight is then a mixture of two Gaussians, so the true reverse step $q(x_s \mid x_t)$ , back to any earlier time $s$ , can be written down. We compare it with the moment-matched Gaussian, the one that has the same mean and variance.

The comparison uses the total variation distance, half the integral of $\lvert p - q\rvert$ . It is the largest difference between the probabilities that the two distributions give to any one event. So 0 means identical and 1 means no overlap at all.

We size a step by how much it changes $\lambda$ , the log of the ratio of the signal's variance to the noise's variance, which this section defines properly below. Large $\lambda$ means nearly clean data. At $\lambda = 0$ the signal and the noise have equal variance, and negative $\lambda$ means mostly noise. The cell starts from noisy points at $\lambda = 0$ , the hardest place, as we will see. It steps back to $\lambda = 0.02$ , then 0.1, and so on up to 8, which removes almost all the noise, and it averages the distance over 100 noisy points.

The cell needs the exact reverse step. In variance preserving form the signal scale is $\alpha = \sqrt{\operatorname{sigmoid}(\lambda)}$ and the noise variance is $\sigma^2 = 1 - \alpha^2$ (both derived below). Write $x_t = a x_s + \sqrt{v}\,z$ with $a = \alpha_t/\alpha_s$ , $v = 1 - a^2$ , and $z$ a standard Gaussian draw. Given $x_t$ and the data point $m = \pm1$ , $x_s$ is Gaussian with precision $P = 1/\sigma_s^2 + a^2/v$ and mean $(\alpha_s m/\sigma_s^2 + a x_t/v)/P$ . The exact reverse step is these two bumps, weighted by how likely each $m$ is given $x_t$ .

Python · cell 1

Starting near $\lambda = 0$ is the hard case, because there the noisy point could still have come from either data point. Much noisier points come from a distribution that is already nearly Gaussian, so their reverse steps are nearly Gaussian too. Much cleaner points are already decided. From $\lambda = 4$ the distance is near zero even for a step all the way to $\lambda = 8$ (try reverse_step_tv(4.0, 8.0)).

The standard schedule's first steps are larger than its average, since $\lambda$ falls by 0.79 between $t = 1$ and $t = 2$ . But they sit near $\lambda = 9$ , where even big steps are harmless for this data (reverse_step_tv(8.42, 9.21) is about $10^{-13}$ ). Real images have fine detail at that scale, which is why §33.2 treats the last reverse step, back to the data itself, separately.

So the reverse process can be learned as a long chain of Gaussian steps. Sohl-Dickstein et al. (2015) proposed exactly this, with a learned mean and variance at each step and a thousand steps for most of their image data. Ho, Jain & Abbeel (2020) fixed the variances, so each step needs only a mean. Learning a mean is regression, fitting a function with squared error. They made the chain produce images that rivaled GANs, under the name denoising diffusion probabilistic models (DDPM). The price is one network call per step, which §33.4 works hard to reduce.

This answers a common interview question. Why are diffusion samples sharp when one-shot generators trained with squared error blur? The reverse of one small step has a single peak, so a Gaussian with a learned mean fits it. The reverse of one big step has several peaks, and a squared-error fit averages them.

The forward process

Everything in this chapter starts from one fixed noising process with no learned parameters. It is a Markov chain, a sequence of random points in which each step depends only on the current point, never on earlier ones. Two letters need care. Here $d$ is the dimension of one data point (the number of pixel values of an image, or 2 for points in the plane), not a model width. And $T$ is the number of noising steps, not a sequence length.

📐 The forward (noising) process

Let $\mathbf{x}_0 \in \R^d$ be drawn from the data distribution $p_{\text{data}}$ . Fix a variance schedule $\beta_1, \dots, \beta_T$ with $0 < \beta_t < 1$ . The forward process is the Markov chain

q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\!\big(\sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\ \beta_t\mathbf{I}\big), \qquad t = 1, \dots, T,

or equivalently $\mathbf{x}_t = \sqrt{1-\beta_t}\,\mathbf{x}_{t-1} + \sqrt{\beta_t}\,\boldsymbol\epsilon_t$ , with $\boldsymbol\epsilon_1, \dots, \boldsymbol\epsilon_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ independent of each other and of $\mathbf{x}_0$ . The joint distribution is $q(\mathbf{x}_{1:T}\mid\mathbf{x}_0) = \prod_{t=1}^T q(\mathbf{x}_t\mid\mathbf{x}_{t-1})$ . The chain has no learned parameters. We write $p_t$ for the marginal density of $\mathbf{x}_t$ , the distribution of the noised data at time $t$ .

In plain words, at every step, shrink the current point toward the origin by a factor slightly below 1 and add a little independent Gaussian noise. The bold $\boldsymbol\epsilon$ is always this standard Gaussian noise. A plain $\epsilon$ keeps its book-wide meaning of a small constant, and the $\beta$ of a $\beta$ -VAE never appears here without a subscript.

Before the general result, here are two steps by hand.

✏️ Two noising steps of one number

Take $x_0 = 2$ , $T = 2$ , $\beta_1 = 0.1$ , and $\beta_2 = 0.2$ . The first step gives $x_1 = \sqrt{0.9}\,x_0 + \sqrt{0.1}\,\boldsymbol\epsilon_1$ . The second gives

x_2 = \sqrt{0.8}\,x_1 + \sqrt{0.2}\,\boldsymbol\epsilon_2 = \sqrt{0.72}\,x_0 + \sqrt{0.08}\,\boldsymbol\epsilon_1 + \sqrt{0.2}\,\boldsymbol\epsilon_2 ,

because $\sqrt{0.8}\sqrt{0.9} = \sqrt{0.72}$ and $\sqrt{0.8}\sqrt{0.1} = \sqrt{0.08}$ . The two noise terms are independent zero-mean Gaussians. So their sum is a zero-mean Gaussian whose variance is the sum of the variances, $0.08 + 0.2 = 0.28$ . Hence

x_2 \mid x_0 \sim \mathcal{N}\big(\sqrt{0.72}\cdot 2,\ 0.28\big) = \mathcal{N}(1.697,\ 0.28).

Notice that $0.28 = 1 - 0.72$ . The signal keeps the fraction $0.72$ of its variance, and the noise fills exactly the rest. If $x_0$ had been random with variance 1 instead of a fixed number, $x_2$ would have variance $0.72 + 0.28 = 1$ , the same as $x_0$ . Two noise draws behave like one draw of variance 0.28.

The example generalizes to every $t$ with a short induction.

∴ The marginal at any time

For the forward process, for every $t = 1, \dots, T$ ,

q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\!\big(\alpha_t\mathbf{x}_0,\ \sigma_t^2\mathbf{I}\big), \qquad \alpha_t^2 = \prod_{s=1}^{t}(1-\beta_s), \qquad \sigma_t^2 = 1 - \alpha_t^2 .

Equivalently $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ with $\boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ independent of $\mathbf{x}_0$ . We set $\alpha_0 = 1$ and $\sigma_0 = 0$ . We call $\alpha_t$ the signal scale and $\sigma_t$ the noise scale. (The DDPM paper writes $\bar\alpha_t$ for our $\alpha_t^2$ . The translation table near the end of this section lists every paper's names.)

Proof. By induction on $t$ . For $t = 1$ the definition gives $\mathbf{x}_1 = \sqrt{1-\beta_1}\,\mathbf{x}_0 + \sqrt{\beta_1}\,\boldsymbol\epsilon_1$ . This is the claim with $\alpha_1^2 = 1 - \beta_1$ and $\sigma_1^2 = \beta_1 = 1 - \alpha_1^2$ . Now suppose $\mathbf{x}_{t-1} = \alpha_{t-1}\mathbf{x}_0 + \sigma_{t-1}\bar{\boldsymbol\epsilon}$ , where $\bar{\boldsymbol\epsilon} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ is built from $\boldsymbol\epsilon_1, \dots, \boldsymbol\epsilon_{t-1}$ and so is independent of $\boldsymbol\epsilon_t$ and $\mathbf{x}_0$ . One more step gives

\mathbf{x}_t = \sqrt{1-\beta_t}\,\alpha_{t-1}\mathbf{x}_0 + \Big(\sqrt{1-\beta_t}\,\sigma_{t-1}\bar{\boldsymbol\epsilon} + \sqrt{\beta_t}\,\boldsymbol\epsilon_t\Big).

The coefficient of $\mathbf{x}_0$ is $\sqrt{1-\beta_t}\,\alpha_{t-1} = \alpha_t$ , by the definition of $\alpha_t^2$ as a product. The bracket is a sum of two independent zero-mean Gaussian vectors with covariances $(1-\beta_t)\sigma_{t-1}^2\mathbf{I}$ and $\beta_t\mathbf{I}$ . So it is a zero-mean Gaussian with covariance

\big((1-\beta_t)\sigma_{t-1}^2 + \beta_t\big)\mathbf{I} = \big((1-\beta_t)(1-\alpha_{t-1}^2) + \beta_t\big)\mathbf{I} = \big(1 - (1-\beta_t)\alpha_{t-1}^2\big)\mathbf{I} = (1 - \alpha_t^2)\mathbf{I} = \sigma_t^2\mathbf{I}.

So the bracket equals $\sigma_t\boldsymbol\epsilon$ for a standard Gaussian $\boldsymbol\epsilon$ independent of $\mathbf{x}_0$ , which is the claim at time $t$ . $\square$

In plain words, a thousand small noise injections add up to one big injection. We can jump straight to any time $t$ with one line, $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ . This is what makes training cheap. To make a training example at time $t$ , there is no need to run $t$ steps of the chain. One draw of $\boldsymbol\epsilon$ is enough, and §33.2 trains on exactly these pairs. Deriving this marginal is one of the most common diffusion interview questions, so practice it until you can do it at a whiteboard.

The same argument, started at time $s$ instead of 0, gives the jump between any two times. The fast samplers of §33.4 use it to skip many steps at once.

∴ From any time to any later time

For $0 \le s < t \le T$ , $q(\mathbf{x}_t \mid \mathbf{x}_s) = \mathcal{N}\big(\alpha_{t\mid s}\mathbf{x}_s,\ \sigma_{t\mid s}^2\mathbf{I}\big)$ with

\alpha_{t\mid s} = \frac{\alpha_t}{\alpha_s}, \qquad \sigma_{t\mid s}^2 = \sigma_t^2 - \alpha_{t\mid s}^2\,\sigma_s^2 = 1 - \frac{\alpha_t^2}{\alpha_s^2}.

For $0 < s < t$ , given $\mathbf{x}_0$ , the correlation between one coordinate of $\mathbf{x}_s$ and the same coordinate of $\mathbf{x}_t$ is $\alpha_{t\mid s}\sigma_s/\sigma_t$ , which is below 1.

∂ Proof of the jump between two times

Steps $s+1, \dots, t$ form a chain of the same kind started at $\mathbf{x}_s$ . So the previous theorem applies with $\mathbf{x}_s$ in place of $\mathbf{x}_0$ . It gives the squared signal scale $\prod_{r=s+1}^t(1-\beta_r) = \alpha_t^2/\alpha_s^2$ and the noise variance $1 - \alpha_t^2/\alpha_s^2$ . The middle form follows from $\sigma_t^2 - (\alpha_t^2/\alpha_s^2)(1 - \alpha_s^2) = 1 - \alpha_t^2/\alpha_s^2$ .

For the correlation, write $\mathbf{x}_t = \alpha_{t\mid s}\mathbf{x}_s + \sigma_{t\mid s}\mathbf{z}$ with fresh noise $\mathbf{z}$ . Given $\mathbf{x}_0$ , each coordinate has $\operatorname{Cov}(x_{s}, x_{t}) = \alpha_{t\mid s}\operatorname{Var}(x_s) = \alpha_{t\mid s}\sigma_s^2$ . Dividing by $\sigma_s\sigma_t$ gives the claim. Since $\sigma_t^2 = \alpha_{t\mid s}^2\sigma_s^2 + \sigma_{t\mid s}^2 > \alpha_{t\mid s}^2\sigma_s^2$ , the correlation is below 1. $\square$

The marginal fixes the distribution at each time, and the jump fixes how two times are linked. A common shortcut in code gets the first right and the second wrong.

⚠️ One noise draw for every time is a different process

The marginal formula holds at each $t$ separately, and the $\boldsymbol\epsilon$ in it is a different random vector at each $t$ . Reusing one $\boldsymbol\epsilon$ for all $t$ still gives every $\mathbf{x}_t$ the right distribution. But it moves each point along a fixed curve from $\mathbf{x}_0$ toward $\boldsymbol\epsilon$ , so $\mathbf{x}_s$ and $\mathbf{x}_t$ have correlation 1 given $\mathbf{x}_0$ , while the true chain has correlation below 1. The widget at the start of this section reuses one $\boldsymbol\epsilon$ on purpose, so that the cloud moves smoothly.

For training the difference is harmless. The training loss looks at one time per example, so any process with the same marginals trains the same network. That observation is also the starting point of a fast sampler that moves without fresh noise, DDIM (Song et al. 2021, §33.4).

Why the chain shrinks the point before adding noise

Why multiply by $\sqrt{1-\beta_t}$ at all, rather than only adding noise? Track the variance of one coordinate when $\mathbf{x}_0$ is random with variance $v_0$ . One step gives $v_t = (1-\beta_t)v_{t-1} + \beta_t$ , which rearranges to

v_t - 1 = (1-\beta_t)(v_{t-1} - 1) \quad\Longrightarrow\quad v_t - 1 = \alpha_t^2\,(v_0 - 1).

So the variance moves geometrically toward 1, and it stays exactly 1 if the data start with variance 1. This is why diffusion models put their data on a scale near 1, images in $[-1, 1]$ (standard deviation about 0.5) and latents rescaled to roughly unit variance. Meanwhile the mean $\alpha_t\mathbf{x}_0$ shrinks to zero. After enough steps $\mathbf{x}_T$ is close to $\mathcal{N}(\mathbf{0},\mathbf{I})$ , whatever $\mathbf{x}_0$ was. That is the point of the shrink. Sampling has to start from a distribution we can draw from without knowing the data, and the shrink makes that distribution the standard Gaussian.

Without the shrink, $\mathbf{x}_t = \mathbf{x}_{t-1} + \sqrt{\beta_t}\,\boldsymbol\epsilon_t$ . The variance becomes $v_0 + \sum_s\beta_s$ and grows without bound, while the mean stays at $\mathbf{x}_0$ forever. The data are never forgotten, only drowned. That process is also used, under the name variance exploding, and we will see shortly that it is the same model in different coordinates.

How close to zero does the signal get? Since $\log(1-\beta) = -\beta - \beta^2/2 - \dots \le -\beta$ ,

\alpha_T^2 = \exp\Big(\sum_{t=1}^T\log(1-\beta_t)\Big) \le \exp\Big(-\sum_{t=1}^T\beta_t\Big),

with near equality when $\sum_t\beta_t^2$ is small.

✏️ How much signal survives a thousand steps

The original DDPM schedule (Ho et al. 2020) raises $\beta_t$ linearly from $10^{-4}$ to $0.02$ over $T = 1000$ steps. The sum of a linear ramp is the number of terms times the average term, $\sum_t\beta_t = 1000\cdot(10^{-4} + 0.02)/2 = 10.05$ . So $\alpha_T^2 \approx e^{-10.05} = 4.3\times10^{-5}$ .

The exact value is $4.04\times10^{-5}$ . The second-order term accounts for the gap, since $\sum_t\beta_t^2/2 = 0.067$ and $e^{-10.05 - 0.067} = 4.04\times10^{-5}$ . The signal scale at the end is $\alpha_T = 0.0064$ , so less than one percent of the original image is left.

A noised Gaussian mixture is still a Gaussian mixture

One more closed form, which the rest of the chapter leans on. A Gaussian mixture is a distribution made of several Gaussian bumps, with bump $k$ chosen with probability $\pi_k$ . The ring of the opening is one, with eight bumps. Most of the chapter's toy data are mixtures, because noise keeps them mixtures.

Proof. Condition on the component $k$ . Then $\alpha_t\mathbf{x}_0 \sim \mathcal{N}(\alpha_t\boldsymbol\mu_k, \alpha_t^2\boldsymbol\Sigma_k)$ and $\sigma_t\boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, \sigma_t^2\mathbf{I})$ are independent Gaussians. So their sum is Gaussian with the summed mean and covariance. Averaging over $k$ with weights $\pi_k$ gives the mixture. $\square$

Because of this theorem, every quantity that later sections need (the best denoiser, the score, the velocity of a flow) has an exact formula for mixture data. So a sampler can be checked against the truth before any network is trained, and every section of this chapter tests its algorithms that way.

Checking the formula by running the chain

The marginal formula is easy to test. The cell runs the actual thousand-step chain on 20,000 copies of $x_0 = 2$ with the DDPM schedule. It compares the empirical mean and variance with $\alpha_t x_0$ and $\sigma_t^2$ at five times.

Python · cell 2

The chain and the formula agree to within sampling error. With 20,000 draws, the typical error is about 0.007 for the mean (the standard deviation divided by $\sqrt{20{,}000}$ ) and 0.01 for the variance. At $t = 1000$ the mean is $2\alpha_T = 0.013$ and the variance is 1, so $x_T$ is a standard Gaussian for all practical purposes.

Now take the ring of the opening, standardized so that each coordinate has variance 1, and look at it at five times.

Python · cell 3

At $t = 50$ the clusters are fatter but still separate. By $t = 200$ they have merged into a ring-shaped haze. There $\alpha_t = 0.81$ , so the noise standard deviation is already 0.58, while neighboring clusters sit $1.08\,\alpha_t = 0.88$ apart. From $t = 500$ on, the picture is a round Gaussian. A reverse process has to put the eight clusters back, and it does most of that work during the steps where they are still separating.

Core

Noise a batch to per-example times

Implement q_sample(x0, t, alpha, sigma, eps), the line that every diffusion training loop starts with. Each example in the batch is noised to its own time, so example $i$ becomes $\alpha_{t_i}\mathbf{x}_{0,i} + \sigma_{t_i}\boldsymbol\epsilon_i$ .

x0 and eps have shape (B, ...), a batch of vectors (B, d) or of images (B, C, H, W).
t is an integer array of shape (B,), with t[i] the time of example $i$ .
alpha and sigma are arrays of length $T + 1$ that hold $\alpha_t$ and $\sigma_t$ at index $t$ , so index 0 is $t = 0$ , the clean data.

Return an array with the same shape as x0.

import numpy as np

def q_sample(x0, t, alpha, sigma, eps):
    return alpha[t] * x0 + sigma[t] * eps        # placeholder: broadcasts along the wrong axis

Tests, verified solutions, and AI help, open once you add your API key.

A schedule is one curve, the signal-to-noise ratio

The schedule $\beta_1, \dots, \beta_T$ looks like a thousand free numbers. It is really one curve. What matters about a noisy point is how much signal it still carries compared with its noise. In the two-step example, after one step the signal had variance 0.9 and the noise 0.1, a ratio of 9. After two steps the ratio was $0.72/0.28 = 2.57$ . That ratio, followed over time, describes the schedule completely.

📐 Signal-to-noise ratio and log-SNR

For a Gaussian corruption $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ , the signal-to-noise ratio and the log-SNR at time $t$ are

\operatorname{SNR}(t) = \frac{\alpha_t^2}{\sigma_t^2}, \qquad \lambda_t = \log\operatorname{SNR}(t),

with the natural log. A schedule must make $\lambda_t$ strictly decreasing in $t$ , from large positive values (almost clean) to large negative ones (almost pure noise). For a variance preserving process, one with $\alpha_t^2 + \sigma_t^2 = 1$ like the forward process above, the log-SNR determines everything,

\alpha_t^2 = \operatorname{sigmoid}(\lambda_t), \qquad \sigma_t^2 = \operatorname{sigmoid}(-\lambda_t),

where $\operatorname{sigmoid}(z) = 1/(1 + e^{-z})$ is the logistic function. The book usually writes it $\sigma(z)$ , but this chapter spells it out because $\sigma_t$ is the noise scale.

The identity takes one line. From $\alpha_t^2/(1 - \alpha_t^2) = e^{\lambda_t}$ we get $\alpha_t^2 = e^{\lambda_t}/(1 + e^{\lambda_t}) = 1/(1 + e^{-\lambda_t})$ , and $\sigma_t^2 = 1 - \alpha_t^2 = 1/(1 + e^{\lambda_t})$ . Hoogeboom et al. (2023) use the same identity to define their schedules directly in $\lambda$ .

In plain words, $\lambda_t = 0$ is the moment when signal and noise have equal variance, and every unit of $\lambda$ is a factor of $e$ in SNR. A variance preserving schedule is completely described by the curve $t \mapsto \lambda_t$ . In the two-step example, $\lambda_1 = \log 9 = 2.20$ and $\lambda_2 = \log 2.57 = 0.94$ .

There is a second convention. Instead of shrinking the data, keep it and let the noise grow.

The two look different. One keeps $\mathbf{x}_t$ near unit scale, and the other lets it grow to $\sigma_{\max}$ . Yet they carry the same information. Take a VP point at $\lambda = 0$ , where $\alpha_t = \sigma_t = 0.707$ and $\mathbf{x}_t = 0.707\,\mathbf{x}_0 + 0.707\,\boldsymbol\epsilon$ . Divide by 0.707 and you get $\mathbf{x}_0 + \boldsymbol\epsilon$ , a VE point at noise level $\sigma = 1 = e^{-0/2}$ .

∴ Only the log-SNR matters

Let $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ for any $\alpha_t > 0$ and $\sigma_t$ , and let $\mathbf{y} = \mathbf{x}_t/\alpha_t$ . Then $\mathbf{y} = \mathbf{x}_0 + \sigma\boldsymbol\epsilon$ is a variance exploding sample at noise level $\sigma = \sigma_t/\alpha_t = e^{-\lambda_t/2}$ . The conditional distribution of $\mathbf{x}_0$ given $\mathbf{x}_t$ equals that of $\mathbf{x}_0$ given $\mathbf{y}$ . In particular, the best denoiser satisfies $\mathbb{E}[\mathbf{x}_0 \mid \mathbf{x}_t = \mathbf{x}] = D(\mathbf{x}/\alpha_t;\ e^{-\lambda_t/2})$ , where $D(\mathbf{y}; \sigma) = \mathbb{E}[\mathbf{x}_0 \mid \mathbf{x}_0 + \sigma\boldsymbol\epsilon = \mathbf{y}]$ is the VE denoiser. Two processes with the same $\lambda_t$ curve differ only by the known rescaling of $\mathbf{x}_t$ by $\alpha_t$ .

Proof. Dividing $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ by $\alpha_t$ gives $\mathbf{y} = \mathbf{x}_0 + (\sigma_t/\alpha_t)\boldsymbol\epsilon$ , and $\sigma_t/\alpha_t = (\alpha_t^2/\sigma_t^2)^{-1/2} = e^{-\lambda_t/2}$ . The map $\mathbf{x}_t \mapsto \mathbf{x}_t/\alpha_t$ is invertible, so observing one is the same as observing the other. Conditioning on either gives the same posterior over $\mathbf{x}_0$ . $\square$

In plain words, the scale of $\mathbf{x}_t$ is bookkeeping. Everything a network could learn about $\mathbf{x}_0$ from $\mathbf{x}_t$ depends on $t$ only through $\lambda_t$ . So one network can be trained in one convention and sampled in another. §33.4 uses this to rewrite sampling in the VE coordinates of one influential design study, EDM (Karras et al. 2022). §33.3 uses it to treat a VE model, the noise-conditional score networks of Song & Ermon (2019), and the VP model of DDPM as one family. An interviewer who asks "VP or VE?" wants to hear that the information is identical and only the numerics differ.

🔬 Which pairs of scales are noising chains

Not every pair of curves $(\alpha_t, \sigma_t)$ comes from a Markov chain that adds noise forward in time. Kingma et al. (2021) give the condition, and it is short. The SNR must never increase.

Let $\mathbf{x}_s = \alpha_s\mathbf{x}_0 + \sigma_s\boldsymbol\epsilon$ for $s < t$ . A Gaussian transition $\mathbf{x}_t = \alpha_{t\mid s}\mathbf{x}_s + \sigma_{t\mid s}\mathbf{z}$ , with $\mathbf{z}$ independent of $\mathbf{x}_0$ and $\boldsymbol\epsilon$ , produces exactly $\mathbf{x}_t \sim \mathcal{N}(\alpha_t\mathbf{x}_0, \sigma_t^2\mathbf{I})$ given $\mathbf{x}_0$ if and only if $\alpha_{t\mid s} = \alpha_t/\alpha_s$ and

\sigma_{t\mid s}^2 = \sigma_t^2 - \frac{\alpha_t^2}{\alpha_s^2}\sigma_s^2 = \alpha_t^2\Big(\frac{1}{\operatorname{SNR}(t)} - \frac{1}{\operatorname{SNR}(s)}\Big).

This variance is nonnegative exactly when $\operatorname{SNR}(t) \le \operatorname{SNR}(s)$ . So a schedule defines a valid noising chain if and only if its SNR never increases.

Proof. Given $\mathbf{x}_0$ , the transition gives mean $\alpha_{t\mid s}\alpha_s\mathbf{x}_0$ and covariance $(\alpha_{t\mid s}^2\sigma_s^2 + \sigma_{t\mid s}^2)\mathbf{I}$ . Matching the mean $\alpha_t\mathbf{x}_0$ for every $\mathbf{x}_0$ forces $\alpha_{t\mid s} = \alpha_t/\alpha_s$ . Matching the covariance forces the stated $\sigma_{t\mid s}^2$ . Factoring out $\alpha_t^2$ gives $\alpha_t^2(\sigma_t^2/\alpha_t^2 - \sigma_s^2/\alpha_s^2)$ , which is the SNR form. $\square$

For the discrete VP chain this recovers $1 - \beta_t = \alpha_t^2/\alpha_{t-1}^2$ . So any strictly decreasing sequence $\alpha_1^2 > \alpha_2^2 > \dots$ in $(0, 1)$ defines valid betas. The cosine schedule below is built this way, by choosing the $\alpha_t^2$ curve first and reading the betas off it.

What the schedule actually controls

If only $\lambda$ matters, what does choosing a schedule do? It makes two separate choices, and it pays to keep them apart, as Dieleman (2024) argues at length.

First, training draws $t$ uniformly from $\{1, \dots, T\}$ . So the schedule decides how often the network practices at each noise level, that is, the distribution of $\lambda$ it sees in training. For example, the cosine schedule with its small offset set to zero spreads its training over $\lambda$ with density $p(\lambda) = \operatorname{sech}(\lambda/2)/(2\pi)$ . That density puts most of the training at moderate $\lvert\lambda\rvert$ and very little at the extremes. At $\lambda = 0$ it is 0.16 per unit of $\lambda$ , and at $\lambda = \pm 10$ it is 0.002.

∂ The density of λ under the cosine schedule

With offset 0, the cosine schedule has $\alpha_t^2 = \cos^2(\pi u/2)$ and $\sigma_t^2 = \sin^2(\pi u/2)$ , with $u = t/T$ . So $\operatorname{SNR} = \cot^2(\pi u/2)$ and $\lambda = -2\log\tan(\pi u/2)$ . Solving for $u$ gives $u = \frac{2}{\pi}\arctan e^{-\lambda/2}$ . Differentiating,

\Big\lvert\frac{\mathrm{d}u}{\mathrm{d}\lambda}\Big\rvert = \frac{2}{\pi}\cdot\frac{\tfrac12 e^{-\lambda/2}}{1 + e^{-\lambda}} = \frac{1}{\pi\,(e^{\lambda/2} + e^{-\lambda/2})} = \frac{\operatorname{sech}(\lambda/2)}{2\pi}.

Since $u$ is uniform on $[0, 1]$ , this is the density of $\lambda$ . It integrates to 1, as a density must.

Second, at sampling time the schedule decides where the reverse steps fall. A training weighting and a sampling grid are different design choices that one schedule happens to tie together. The bound on the likelihood does not care about the shape of the curve. The variational bound is a lower bound on the log-likelihood, the ELBO of §32.2, which §33.2 writes out for diffusion and trains on. In the limit of infinitely many steps it depends on the schedule only through the two endpoints $\lambda_{\min}$ and $\lambda_{\max}$ (Kingma et al. 2021), as that section shows. So the shape of the curve matters only through these two roles, and §33.7 compares the resulting weightings on one axis.

The VP and VE conventions are not the only ones. Rectified flow models move noise to data along straight lines, and they power the open text-to-image model Stable Diffusion 3. They use the linear path $\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\boldsymbol\epsilon$ for continuous $t \in [0, 1]$ . So $\alpha_t = 1 - t$ , $\sigma_t = t$ , and $\lambda_t = 2\log\big((1-t)/t\big)$ . At $t = \tfrac12$ the signal and noise scales are both $\tfrac12$ , so $\lambda = 0$ and $\alpha_t^2 + \sigma_t^2 = \tfrac12$ . The path is neither VP nor VE. Yet by the theorem above it carries the same information as a VP process with the same $\lambda_t$ . §33.6 builds its models on this path.

Three schedules, in numbers

Three discrete schedules are the ones you will meet most often in papers and code. We write all three with $T = 1000$ .

Linear, from DDPM (Ho et al. 2020). $\beta_t$ rises linearly from $\beta_1 = 10^{-4}$ to $\beta_T = 0.02$ . The authors chose values that are small next to data scaled to $[-1, 1]$ , so that each reverse step has nearly the same Gaussian form as the forward step.
Cosine, from a follow-up paper that improved DDPM's likelihoods and sampling speed (Nichol & Dhariwal 2021). It defines the signal directly, $\alpha_t^2 = f(t)/f(0)$ with $f(t) = \cos^2\!\big(\frac{t/T + s}{1 + s}\cdot\frac{\pi}{2}\big)$ and offset $s = 0.008$ . Then it sets $\beta_t = 1 - \alpha_t^2/\alpha_{t-1}^2$ and clips it at $0.999$ . The offset keeps the first steps from being too small, because the authors found that tiny noise at the start made the network's noise predictions hard to get accurate. With $s = 0.008$ the first step's noise standard deviation, $\sqrt{\beta_1} = 0.0064$ , sits just below one pixel bin of an 8-bit image scaled to $[-1, 1]$ , which is $1/127.5 = 0.0078$ .
Scaled linear, from Stable Diffusion. $\sqrt{\beta_t}$ rises linearly from $\sqrt{0.00085}$ to $\sqrt{0.012}$ . Stable Diffusion 1 and 2 and its larger successor SDXL use it, in their released scheduler configurations. All three are open text-to-image models that diffuse in the latent space of an autoencoder (§32.1), a compressed grid of numbers that the autoencoder decodes into pixels.

The cell builds all three as arrays indexed by $t = 0, \dots, T$ and prints their key numbers. The next cell plots them.

Python · cell 4

Python · cell 5

Read the plot and the last two columns of the printed table together. The linear schedule reaches equal signal and noise at $t = 259$ . It then spends 30 percent of all its steps at $\lambda < -5$ , where the SNR is below 0.007 and the image is essentially gone. Nichol & Dhariwal (2021) observed the consequence directly. They trained with the linear schedule on ImageNet, a dataset of photos, at $64\times64$ . Then they skipped the first part of the reverse process, its noisiest steps, and measured sample quality with FID, the Fréchet Inception distance. FID compares the statistics of an image classifier's features on generated images and on real ones, and lower is better. Skipping up to 20 percent of the reverse process barely raised FID (their Figure 4), so those steps contribute little.

The cosine schedule was designed so that $\alpha_t^2$ falls roughly linearly in the middle and changes slowly at both ends. It reaches SNR 1 near the midpoint, at $t = 496$ , and spends only 5 percent of its steps below $\lambda = -5$ .

The cosine schedule has its own quirk, the cliff at the right end of the $\lambda$ plot. Without the clip its last step would be $\beta_T = 1 - f(T)/f(T-1) = 1$ , because $f(T) = \cos^2(\pi/2) = 0$ . Only that last step is clipped. So $\lambda$ falls from $-12.93$ at $t = 999$ to $-19.84$ at $t = 1000$ in a single step. That jump of 6.9 is far larger than any other step of the three schedules, the largest of which is 1.39.

Warm-up

Build the linear and cosine schedules

Implement make_schedule(T, kind) for two kinds of schedule.

"linear", the DDPM schedule, with $\beta_t$ linear from $10^{-4}$ to $0.02$ .
"cosine", the cosine schedule, with offset $s = 0.008$ and $\beta_t$ clipped at 0.999.

Return (alpha, sigma, lam), three arrays of length $T + 1$ . Index $t$ holds $\alpha_t$ , $\sigma_t$ , and $\lambda_t = \log(\alpha_t^2/\sigma_t^2)$ for $t = 0, \dots, T$ . Index 0 is the clean data, with $\alpha_0 = 1$ , $\sigma_0 = 0$ , and $\lambda_0 = +\infty$ .

import numpy as np

def make_schedule(T, kind):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Changing the number of steps

A schedule written as $\beta_1, \dots, \beta_T$ is tied to its $T$ . Suppose we want the same $\lambda$ curve, as a function of the fraction $t/T$ , with a different number of steps. Since $\log\alpha_t^2 \approx -\sum_{s \le t}\beta_s$ , the partial sums $\sum_{s \le t}\beta_s$ must stay the same at matching fractions. So each $\beta_t$ must scale like $1/T$ . Doubling the number of steps halves every beta.

Nichol & Dhariwal (2021) did exactly this when they trained with $T = 4000$ . They ran $\beta_t$ from $0.0001/4$ to $0.02/4$ to keep the shape of $\alpha_t^2$ . The same picture says that a schedule is really a continuous rate, $\beta(u) = T\beta_t$ at $u = t/T$ . That rate is how the chain is written as a stochastic differential equation, its limit as the steps become infinitely many and infinitely small, in §33.3.

Research

How many steps does a schedule need?

This exercise asks how many steps a linear ramp of betas needs to reach a given terminal SNR, and how to change $T$ without changing the curve. Implement three functions. In the first two, target is a terminal SNR between 0 and 1.

approx_min_steps(beta_1, beta_T, target) keeps the endpoints $\beta_1$ and $\beta_T$ fixed and stretches the ramp over $T$ steps. It returns the first-order estimate of the smallest $T$ with $\operatorname{SNR}(T) \le$ target, using $\operatorname{SNR}(T) \approx \alpha_T^2 \approx \exp(-\sum_t\beta_t)$ , as an integer rounded up.
min_steps(beta_1, beta_T, target) returns the exact smallest such $T$ . It must agree with a brute-force scan, but it should start from your estimate rather than from 1.
rescaled_linear_betas(T, T_ref=1000, beta_1=1e-4, beta_T=0.02) returns a linear ramp of $T$ betas whose $\lambda$ curve, as a function of $t/T$ , approximately matches the curve of the linear ramp from beta_1 to beta_T over T_ref steps. Scale both endpoints by one common factor, the one the first-order argument above gives. Here the endpoints change with $T$ .

import numpy as np

def approx_min_steps(beta_1, beta_T, target):
    raise NotImplementedError

def min_steps(beta_1, beta_T, target):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

Terminal SNR, and what breaks when it is not zero

Sampling starts from $\mathbf{x}_T \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , pure noise. Training, though, only ever shows the network $\mathbf{x}_T = \alpha_T\mathbf{x}_0 + \sigma_T\boldsymbol\epsilon$ , which still contains a little signal unless $\alpha_T = 0$ . The terminal SNR, $\operatorname{SNR}(T)$ , measures this gap between what the network learned to expect at $t = T$ and what it is given when sampling.

For DDPM's linear schedule the leak is tiny. The KL divergence $D_{\mathrm{KL}}(q\Vert p)$ measures how far one distribution is from another, in the expected log-ratio of their densities (§0.5). Ho et al. (2020) report that the KL between the last noisy point $q(\mathbf{x}_T\mid\mathbf{x}_0)$ and pure noise was about $10^{-5}$ bits per dimension, that is, per pixel value. For Stable Diffusion's schedule the leak is not tiny. At $t = T$ ,

\mathbf{x}_T = 0.068\,\mathbf{x}_0 + 0.998\,\boldsymbol\epsilon .

✏️ The brightness leak in Stable Diffusion

A signal at 7 percent of its scale is hard to see in one pixel, but the network sees all of them at once. Stable Diffusion at $512\times512$ diffuses a latent with four channels of $64\times64$ numbers each. Average one channel over its 4,096 positions.

The noise in that average has standard deviation $0.998/\sqrt{4096} = 0.0156$ .
The signal in it is $0.068$ times the channel mean $m$ of the clean latent.
The SNR of the channel mean is therefore $(0.068\,m/0.0156)^2 \approx 19\,m^2$ . For a channel mean of $m = 0.5$ that is about 5, plainly readable.

So the overall brightness and color of the image are visible in $\mathbf{x}_T$ during training, and the network learns to respect them. When sampling, though, the channel means of pure noise are $0 \pm 0.0156$ . The network is told every time that the image has average brightness.

Lin et al. (2023) traced exactly this failure in Stable Diffusion. It always generates images of medium brightness, and it fails prompts as plain as "Solid black color" or "A white background".

⚠️ A schedule that never reaches pure noise

A nonzero terminal SNR is a mismatch between training and sampling. Lin et al. (2023) proposed four fixes, and they come as a package.

Rescale the schedule so that $\operatorname{SNR}(T) = 0$ .
Train with v-prediction. A network usually predicts the noise $\boldsymbol\epsilon$ (§33.2). At $\alpha_T = 0$ its input is $\mathbf{x}_T = \boldsymbol\epsilon$ , so predicting $\boldsymbol\epsilon$ is trivial. But recovering the data from the network's noise guess $\hat{\boldsymbol\epsilon}$ , as $\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sigma_t\hat{\boldsymbol\epsilon})/\alpha_t$ , divides by zero. The v target, $\alpha_t\boldsymbol\epsilon - \sigma_t\mathbf{x}_0$ , is $-\mathbf{x}_0$ at $t = T$ . It asks the network for the data itself, up to sign, so the network learns the mean image for the prompt (§33.4 treats the v target in full).
Make the sampler start from $t = T$ itself. Several popular samplers never visit the last timestep (§33.4 explains this trailing-timestep issue).
Rescale classifier-free guidance, the sampling trick that pushes samples toward the prompt by extrapolating from a prediction without the prompt to one with it. Without the rescale it overexposes images once the terminal SNR is zero (§33.5).

The first fix is a short formula. It keeps $\alpha_1$ , sends $\alpha_T$ to zero, and stretches everything in between linearly in $\alpha_t$ .

Lin et al. report that rescaling in $\alpha_t$ , the square root of DDPM's $\bar\alpha_t$ , preserves the shape of the curve better than rescaling the SNR. For the cosine schedule no rescaling is needed. Deleting the clip already gives $\beta_T = 1$ .

A variance exploding process cannot reach zero terminal SNR at all, since $\sigma_{\max}$ is finite. That is one reason Lin et al. recommend the variance preserving form. VE models instead make $\sigma_{\max}$ large compared with the distances between data points, so that the remaining signal is negligible. In an interview, terminal SNR is the standard example of a mismatch between training and testing hidden inside a noise schedule.

Interview

Rescale a schedule to zero terminal SNR

Implement zero_terminal_snr(beta), the rescaling of Lin et al. (2023) defined above. You are given an array of betas $\beta_1, \dots, \beta_T$ .

Compute $\alpha_t = \sqrt{\prod_{s \le t}(1 - \beta_s)}$ for $t = 1, \dots, T$ .
Shift and scale it so that $\alpha_1$ is unchanged and $\alpha_T = 0$ .
Convert back and return the new betas, an array of length $T$ .

Do not modify the input array.

import numpy as np

def zero_terminal_snr(beta):
    beta = beta.copy()
    beta[-1] = 1.0            # placeholder: forces alpha_T = 0 but leaves the rest of the schedule alone
    return beta

Tests, verified solutions, and AI help, open once you add your API key.

Why bigger images need more noise

A larger image needs more noise to hide the same amount of its content. Take a schedule that works well for $64\times64$ images and train at $512\times512$ . The same $\lambda_t$ now hides much less of the image. Neighboring pixels of a natural image are strongly correlated, and the noise in them is not. So a network that looks at a patch of pixels can average the noise away and still see the layout of the image. Chen (2023) shows noised images at equal noise level and several resolutions, and the large ones are visibly easier to read.

The averaging argument is exact in the simplest case. Let the image be constant, every one of its $n$ pixels equal to $c$ , and noise it as $x_{t,i} = \alpha_t c + \sigma_t\boldsymbol\epsilon_i$ . The average of the $n$ noisy pixels is $\alpha_t c$ plus noise of standard deviation $\sigma_t/\sqrt n$ . So the SNR of the image's one true number is $n\,\alpha_t^2/\sigma_t^2$ , or in log terms $\lambda_t + \log n$ . At $\lambda_t = 0$ a single pixel has SNR 1, but the average of a flat $64\times64$ patch has SNR 4,096. For a real image, average-pooling a $k\times k$ block leaves the smooth part of the signal nearly unchanged and divides the noise variance by $k^2$ .

📌 More pixels need more noise

Averaging $k^2$ pixels whose signal is the same and whose noise is independent multiplies the SNR by $k^2$ . So the coarse structure of an image with side $L$ has an effective log-SNR of about $\lambda_t + 2\log(L/L_{\text{ref}})$ , compared with the same image at side $L_{\text{ref}}$ . Keeping the coarse structure as hidden as at the reference resolution requires shifting the schedule to $\lambda_t - 2\log(L/L_{\text{ref}})$ . Hoogeboom et al. (2023) shift by exactly this in their pixel-space model, simple diffusion, adding $2\log(64/L)$ with $64\times64$ as reference. Stable Diffusion 3 shifts its timesteps so that $\lambda$ drops by $2\log\alpha$ , with the shift $\alpha = \sqrt{m/n}$ (a number, not the signal scale $\alpha_t$ ) for an image of $m$ pixels and a reference of $n$ pixels. In practice it uses $\alpha = 3.0$ at $1024\times1024$ , below the formula's 4 for its $256\times256$ pretraining resolution, which lowers $\lambda$ by $2\log 3 = 2.20$ (Esser et al. 2024).

The cell checks the argument on a smooth synthetic picture rendered at several sizes. The per-pixel SNR is 1 at every size. The cell then average-pools each noisy picture down to $8\times8$ . It computes the SNR of that small picture from the pooled clean signal and the exact pooled noise variance $\sigma_t^2/k^2$ .

Python · cell 6

Without a shift, the coarse SNR grows from 1.0 at $8\times8$ to 869 at $256\times256$ . That is $32^2 = 1024$ times the 0.85 of the signal variance that survives pooling. With the shift, it stays near 0.85 for every size above $8\times8$ . A model trained at $256\times256$ with a schedule tuned at $8\times8$ would spend nearly all of its training at noise levels where the layout of the image is already obvious. It would barely practice the hard part, deciding the layout from noise.

The widget shows the same effect on a picture. Choose the number of pixels per side and the per-pixel noise scale. The three panels show the clean picture, the noisy one, and the noisy one averaged down to $8\times8$ , roughly what the coarse layout looks like to the network. The big number is the SNR of that $8\times8$ view. Keep the noise fixed, raise the size, and watch the layout come through and the number grow. The "shift by size" button lowers $\lambda$ by $2\log(n/32)$ , with $32\times32$ as the reference, and the number then barely changes with size.

On the linear path of rectified flow, Stable Diffusion 3 applies the shift to the time itself. For an image of $m$ pixels, a time $t$ tuned for $n$ pixels becomes

t_m = \frac{\alpha\,t}{1 + (\alpha - 1)\,t}, \qquad \lambda_{t_m} = \lambda_t - 2\log\alpha, \qquad \alpha = \sqrt{m/n}.

Here $\alpha$ is the shift, not a signal scale. In plain words, the shift maps each timestep to a later, noisier one, keeps $t = 0$ and $t = 1$ fixed, and lowers the log-SNR by the same constant everywhere. With $\alpha = 3$ , the midpoint $t = 0.5$ moves to $0.75$ .

∂ The timestep shift from the constant-image argument

Esser et al. (2024) derive the shift from the same constant image. Take $\mathbf{x}_t = (1-t)c\mathbf{1} + t\boldsymbol\epsilon$ on $n$ pixels. Dividing the pixel average by $1 - t$ estimates $c$ with standard deviation $\frac{t}{1-t}\cdot\frac{1}{\sqrt n}$ .

Ask for equal uncertainty at $n$ pixels and time $t_n$ and at $m$ pixels and time $t_m$ . That gives $\frac{t_m}{1 - t_m} = \alpha\,\frac{t_n}{1 - t_n}$ with $\alpha = \sqrt{m/n}$ . Solving for $t_m$ gives $t_m = \alpha t_n/(1 + (\alpha - 1)t_n)$ . Since $\lambda_t = 2\log\frac{1-t}{t}$ on the linear path, multiplying $\frac{t}{1-t}$ by $\alpha$ lowers $\lambda$ by exactly $2\log\alpha$ .

The constant image is not a realistic picture, and the authors say so. They pretrained at $256\times256$ , so the formula would give $\alpha = 4$ at $1024\times1024$ . Instead they tried several shifts when sampling from a model trained at $1024\times1024$ and ran a human preference study. Raters strongly preferred shifts above 1.5 and showed little difference among the larger ones. The authors used $\alpha = 3.0$ , a value from that range, for both training and sampling at $1024\times1024$ .

Chen (2023) found the same trend, that the best schedule gets noisier as images get larger because neighboring pixels repeat each other. He also studies a second way to add noise, taken from his group's earlier work. Scale the data by a factor $b < 1$ before noising. That multiplies the SNR by $b^2$ , which moves the whole $\lambda$ curve down by $2\log(1/b)$ , the same move as a shifted schedule. The noised input then has variance below 1, so it is renormalized to unit variance before it enters the network.

Latent spaces do not remove the issue. An autoencoder that downsamples by 8 in each direction (§32.1) leaves a latent with less redundancy per element than raw pixels. Yet Stable Diffusion 3 diffuses in such a latent and still needs its shift at high resolution, and latent diffusion returns to this. An interviewer may ask why a model fine-tuned at a higher resolution needs a new schedule. The answer is this averaging argument and the shift by $2\log$ of the size ratio.

Warm-up

The resolution shift for the linear path

Implement three functions, each of which accepts scalars or arrays.

logsnr_linear_path(t) returns the log-SNR $\lambda_t$ of $\mathbf{x}_t = (1-t)\mathbf{x}_0 + t\boldsymbol\epsilon$ .
shift_timesteps(t, shift) returns the Stable Diffusion 3 map $t \mapsto \alpha t/(1 + (\alpha - 1)t)$ , with $\alpha$ = shift.
resolution_shift(m, n) returns the shift $\alpha$ for a schedule tuned at $n$ pixels and used at $m$ pixels.

The tests check numerically that the shift lowers $\lambda$ by exactly $2\log\alpha$ .

import numpy as np

def logsnr_linear_path(t):
    raise NotImplementedError

def shift_timesteps(t, shift):
    raise NotImplementedError

Tests, verified solutions, and AI help, open once you add your API key.

🔬 Noise drowns the fine detail first

There is a frequency picture of the same effect. The power spectrum of natural images falls off roughly as a power law in spatial frequency, while white Gaussian noise has equal power at every frequency. So noise of a given level buries the high frequencies (fine texture) first and the low frequencies (layout) last, and the reverse process rebuilds coarse structure before fine detail. Dieleman (2024) develops this reading in his essay "Diffusion is spectral autoregression", where diffusion becomes an approximate coarse-to-fine autoregression over frequencies. It is an interpretation that fits measured spectra, not a theorem. But it explains both the resolution shift (more pixels add high frequencies, not low ones) and why image diffusion models are said to decide composition early in sampling.

Choosing a schedule

The table collects the schedules of this section, with the fixes of the last two parts and the linear path of §33.6.

schedule	definition	$\lambda_1$ and $\lambda_T$ at $T = 1000$	$\operatorname{SNR}(T)$	used by	weakness
linear	$\beta_t$ linear from $10^{-4}$ to $0.02$	9.21, $-10.12$	$4.0\times10^{-5}$	DDPM	30 percent of steps have $\lambda_t < -5$ , wasted at low resolution
cosine	$\alpha_t^2 = f(t)/f(0)$ , $s = 0.008$ , $\beta_t \le 0.999$	10.09, $-19.84$	$2.4\times10^{-9}$	Improved DDPM	the clipped last step jumps $\lambda$ by 6.9
scaled linear	$\sqrt{\beta_t}$ linear from $\sqrt{0.00085}$ to $\sqrt{0.012}$	7.07, $-5.36$	0.0047	Stable Diffusion 1 and 2, SDXL	terminal SNR far from 0, a medium-brightness bias
zero terminal SNR	any of the above with $\alpha_t$ rescaled so $\alpha_T = 0$	$\lambda_1$ unchanged, $-\infty$	0	Lin et al. 2023	the $\boldsymbol\epsilon$ target is useless at $t = T$ , so it needs v-prediction
shifted	$\lambda$ moved down by $2\log$ of the resolution ratio	shifted by a constant	smaller	simple diffusion, Stable Diffusion 3	the shift is tuned by hand, not derived exactly
linear path	$\alpha_t = 1 - t$ , $\sigma_t = t$ for $t \in [0, 1]$ (not VP)	$+\infty$ , $-\infty$	0	Stable Diffusion 3, FLUX.1	trains best with a nonuniform distribution over $t$

Translating between the papers' notations

Diffusion papers use at least five notations for the same few quantities. The most common bug when implementing a paper is reading its $\alpha_t$ as someone else's. The table below translates the papers this chapter uses.

DDPM, the discrete chain of this section (Ho et al. 2020), and Improved DDPM, which added the cosine schedule (Nichol & Dhariwal 2021).
DDIM, a deterministic sampler for the same trained network (Song et al. 2021, §33.4).
VDM, variational diffusion models written in continuous time and log-SNR (Kingma et al. 2021), and DPM-Solver, a fast solver for the sampling equation (Lu et al. 2022).
EDM, the design study that writes everything in the VE noise level (Karras et al. 2022).
SD3, Stable Diffusion 3, on the linear path (Esser et al. 2024).
Flow matching (Lipman et al. 2023) and rectified flow (Liu et al. 2023), which learn a velocity that carries noise to data (§33.6).

This book uses one notation in every section of the chapter, with time running from data at $t = 0$ to noise at $t = T$ (discrete) or $t = 1$ (continuous), flow matching included.

📐 Reading the papers, one notation for the chapter

this book	DDPM, Improved DDPM	DDIM	VDM, DPM-Solver	EDM	SD3	Lipman et al. flow matching, Liu et al. rectified flow
$\alpha_t^2$	$\bar\alpha_t$	$\alpha_t$	$\alpha_t^2$	$s(t)^2$	$a_t^2$	(time reversed)
$1 - \beta_t = \alpha_t^2/\alpha_{t-1}^2$	$\alpha_t$	$\alpha_t/\alpha_{t-1}$	$\alpha_{t\mid t-1}^2$
$\sigma_t$	$\sqrt{1 - \bar\alpha_t}$	$\sqrt{1 - \alpha_t}$	$\sigma_t$	$s(t)\,\sigma(t)$	$b_t$
$\lambda_t = \log(\alpha_t^2/\sigma_t^2)$			VDM writes $\operatorname{SNR} = e^{-\gamma(t)}$ , so $\gamma = -\lambda$ ; DPM-Solver's $\lambda_t$ is half of ours	$-2\log\sigma$	$\lambda_t$ , same as ours
VE noise level $\sigma = \sigma_t/\alpha_t$				$\sigma(t)$
data at $t = 0$ , noise at the end	same	same	same	$\sigma$ grows with $t$	same	data at $t = 1$ , noise at $t = 0$
linear path $(1-t)\mathbf{x}_0 + t\boldsymbol\epsilon$ , velocity $\boldsymbol\epsilon - \mathbf{x}_0$					same	$t\mathbf{x}_1 + (1-t)\mathbf{x}_0$ with $\mathbf{x}_1$ data and $\mathbf{x}_0$ noise, velocity data minus noise
reverse-step variance $\nu_t^2$	$\sigma_t^2$	$\sigma_t(\eta)^2$

A blank cell means the paper has no separate symbol for that quantity. In EDM the noised sample is $\mathbf{x} = s(t)\,\hat{\mathbf{x}}$ , where $\hat{\mathbf{x}}$ is data plus noise of level $\sigma(t)$ . That is why its scale $s(t)$ plays the role of our $\alpha_t$ . The reverse-step variance $\nu_t^2$ is the variance of the noise a sampler adds at each reverse step (§33.2). It gets its own letter because DDPM's and DDIM's $\sigma_t$ name that sampling noise, not the forward noise scale.

⚠️ Two papers, two alphas, two directions of time

In DDPM, $\alpha_t = 1 - \beta_t$ is the one-step keep factor and $\bar\alpha_t = \prod_{s\le t}\alpha_s$ is the cumulative signal variance. DDIM renames DDPM's $\bar\alpha_t$ to $\alpha_t$ . VDM and DPM-Solver use $\alpha_t$ for the cumulative signal scale, the square root of DDPM's $\bar\alpha_t$ , which is our convention. So "alpha" means a one-step factor, a cumulative variance, or a cumulative scale, depending on the paper. A DDIM update fed DDPM's per-step $\alpha_t$ array produces garbage that still looks like noise, so nothing crashes.

Time direction is the second trap. Lipman et al. and Liu et al. put data at $t = 1$ and noise at $t = 0$ , so their velocity points from noise to data. SD3 and this book put noise at $t = 1$ . Before implementing a formula, write down what the paper's symbols are at both ends of time.

In PyTorch

Production code keeps the schedule in a small module of float64 buffers. It gathers the values for each example and casts them to the data's dtype only at the end. The block builds the three schedules, noises a batch of latents, applies the zero terminal SNR rescaling, and shows why schedules should not be stored in half precision.

🔥 In PyTorchfor a machine with PyTorch

The cosine schedule's $\alpha_T^2 = 2.43\times10^{-9}$ is below the smallest positive float16 number, about $6\times10^{-8}$ . So a schedule stored in half precision silently has $\alpha_T = 0$ and $\lambda_T = -\infty$ . That is exactly the regime where noise prediction breaks, as the terminal SNR part showed. The zero-SNR rescaling leaves $\lambda_1$ at 7.07 and lowers the middle of Stable Diffusion's curve only slightly ( $\lambda_{500}$ moves from $-0.96$ to $-1.14$ ). It sends $\alpha_T$ from 0.068 to exactly 0.

🧭 What to remember

One-shot denoising returns a conditional mean, which blurs. The reverse of a small Gaussian noising step is itself nearly Gaussian, so creation can be learned as a long chain of easy regressions. On two-point data, a reverse step of 0.02 in log-SNR is $1.5\times10^{-4}$ from a Gaussian in total variation.
The forward process $q(\mathbf{x}_t\mid\mathbf{x}_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}\,\mathbf{x}_{t-1}, \beta_t\mathbf{I})$ has the closed-form marginal $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ , with $\alpha_t^2 = \prod_{s\le t}(1-\beta_s)$ and $\sigma_t^2 = 1 - \alpha_t^2$ . So a training example at any time costs one noise draw.
Only the log-SNR $\lambda_t = \log(\alpha_t^2/\sigma_t^2)$ matters. VP has $\alpha_t^2 = \operatorname{sigmoid}(\lambda_t)$ , VE is the same process divided by $\alpha_t$ with $\sigma = e^{-\lambda_t/2}$ , and a schedule is a valid noising chain exactly when its SNR never increases.
DDPM's linear schedule runs $\lambda$ from 9.21 to $-10.12$ and wastes 30 percent of its steps below $\lambda = -5$ . The cosine schedule runs from 10.09 to $-19.84$ . Stable Diffusion's scaled linear schedule stops at $\operatorname{SNR}(T) = 0.0047$ .
A nonzero terminal SNR leaks the image's mean brightness into training but not into sampling. The fix is a rescaled schedule with $\alpha_T = 0$ , v-prediction, sampling from $t = T$ , and rescaled guidance.
Larger images need more noise, because averaging $k^2$ pixels multiplies the SNR of coarse structure by $k^2$ . Shift $\lambda$ by $-2\log$ of the resolution ratio, as simple diffusion and Stable Diffusion 3 do (SD3 uses the shift $\alpha = 3$ at $1024^2$ , which lowers $\lambda$ by 2.20).
Papers disagree on what $\alpha_t$ means and which end of time is noise, so translate every formula into this chapter's notation before using it. Next, §33.2 derives the reverse process and the loss that trains it.

Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. The paper that introduced diffusion models, with the small-step argument for why the reverse process has the same Gaussian form.
Ho, Jain & Abbeel (2020). Denoising Diffusion Probabilistic Models. The linear schedule, the closed-form marginal in the notation most code uses, and the results that made the field take diffusion seriously.
Nichol & Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models. Section 3.2 introduces the cosine schedule and shows how much of the linear schedule is wasted.
Lin, Liu, Li & Yang (2023). Common Diffusion Noise Schedules and Sample Steps are Flawed. Terminal SNR, the brightness bug in Stable Diffusion, and the rescaling algorithm. Short and very practical.
Chen (2023). On the Importance of Noise Scheduling for Diffusion Models. An empirical study showing the best schedule gets noisier with image size, and the alternative of scaling the data down before noising.
Hoogeboom, Heek & Salimans (2023). Simple diffusion: End-to-end diffusion for high resolution images. The shifted cosine schedule, defined in log-SNR, with clean ablations at 128 and 256 pixels.
Esser et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Stable Diffusion 3. Section 5.3.2 derives the resolution-dependent timestep shift used in most current image models.
Dieleman (2024). Noise schedules considered harmful. A clear essay on why a schedule is really two separate choices, a training weighting over noise levels and a sampling discretization.

Made for larger screens

Creating by denoising

Why many small steps are easier than one big one

The forward process

Why the chain shrinks the point before adding noise

A noised Gaussian mixture is still a Gaussian mixture

Checking the formula by running the chain

Noise a batch to per-example times

A schedule is one curve, the signal-to-noise ratio

What the schedule actually controls

Three schedules, in numbers

Build the linear and cosine schedules

Changing the number of steps

How many steps does a schedule need?

Terminal SNR, and what breaks when it is not zero

Rescale a schedule to zero terminal SNR

Why bigger images need more noise

The resolution shift for the linear path

Choosing a schedule

Translating between the papers' notations

In PyTorch

Read more

Watch next