This section is about testing diffusion models and the claims people make about them. It shows how to decide whether one method really beats another, how to measure samples when the true distribution is known, how to catch a model that copies its training data, and how to find the bug when samples come out wrong. At the end it carries diffusion from images to text. These are the skills a lab expects when you report a result or review someone else's, and interviewers ask about them right after the derivations.

Here is the kind of claim the section teaches you to check. A colleague trains two generative models on the same 2D data, with the same network and the same training budget. One predicts the noise that was added to a data point, as the denoising diffusion probabilistic model (DDPM) of §33.2 does. The other predicts a velocity along a straight path from data to noise, as the flow matching of §33.6 does. Each model draws samples with 8 calls of its network, on its usual time grid. Averaged over five training runs, 29 percent of the noise predictor's samples land off the data, against 21 percent for flow matching. Is flow matching the better method?

Hold the answer for one more measurement. Replace both networks by the exact denoiser of the data, the best function any training could produce, and run the same two samplers with the same 8 calls. Now the first puts 10.6 percent of its samples off the data and the second 5.2 percent. Nothing was learned, so the only difference left is where each sampler places its 8 steps. Yet one sampler is already twice as accurate as the other. Before the method gets the credit, we need to know how much of the gap comes from the time grid, how much from the network, and how much from the luck of one training run.

We answer that in steps. First we split a sample's error into the sampler's part and the network's part. Then we show that every training loss of the chapter is one loss with a different weight on each noise level. We build distances between sets of samples, run a fair comparison with confidence intervals, test for copying, and collect the debugging checks that save weeks. The last part builds diffusion for text, the masked diffusion language models that now compete with left-to-right decoding.

Here is the setting, restated so the section stands on its own. A data point is $\mathbf{x}_0 \in \R^d$, with $d = 2$ in this lab. Time runs from data at $t = 0$ to pure noise at $t = 1$. The noisy point at time $t$ is a shrunken copy of the data plus Gaussian noise (§33.1),

where the signal scale $\alpha_t$ falls and the noise scale $\sigma_t$ rises as $t$ grows. One number says how noisy $\mathbf{x}_t$ is, the log signal-to-noise ratio $\lambda_t = \log(\alpha_t^2/\sigma_t^2)$, or log-SNR. At $\lambda = 0$ signal and noise have equal power. At $\lambda = 4$ the signal has $e^4 \approx 55$ times the power of the noise, and at $\lambda = -4$ the noise has 55 times the power of the signal.

The data are the chapter's 8-Gaussian ring. It has eight equally likely clusters, called modes, with centers $\boldsymbol\mu_1, \dots, \boldsymbol\mu_8$ on a circle of radius 1.411. Each mode is a round Gaussian with standard deviation $s = 0.0705$, and the scale is chosen so that each coordinate has unit variance. The exact denoiser $\mathbb{E}[\mathbf{x}_0\mid\mathbf{x}_t]$ is the average of all the clean points that could have produced the noisy point $\mathbf{x}_t$, each weighted by how likely it is. It is the best possible prediction of $\mathbf{x}_0$ under squared error (§33.2). A noised Gaussian mixture is again a Gaussian mixture, so for the ring the exact denoiser, the score, and the velocity all have closed forms. That is what makes an honest lab possible. We can always compare a trained network with the truth.

A sample is wrong for three reasons

A sample from a trained diffusion model can be wrong for three separate reasons, and each one has its own fix.

1. Discretization. A sampler follows a continuous path from noise to data in a finite number of steps. The deterministic sampler of §33.4, called DDIM, follows the probability flow ODE of §33.3, the differential equation that carries noise to data. It takes Euler steps, each of which moves in a straight line along the current direction. On a curved path that leaves an error, which shrinks as the steps shrink.
2. Estimation. The network is not the exact denoiser. Its capacity and its training are finite, and its error differs from one noise level to another.
3. Weighting. Training spreads limited capacity over the noise levels. How much the loss charges for an error at each level decides which levels the network gets right. This choice changes the network's errors without changing the function it aims at.

A fourth error lives outside the model. Every metric is computed from finitely many samples, so it has noise and blind spots of its own.

The first two errors can be separated with one trick. Run the sampler twice, once calling the network and once calling the exact denoiser.

In plain words, run the sampler once with the network and once with the truth. The second run shows the error the sampler makes on its own. Whatever is left over belongs to the network. On a Gaussian mixture the oracle is a few lines of NumPy, so the split costs nothing.

The bound needs a distance with the triangle inequality, which says that the direct distance between two distributions is never longer than a detour through a third. Two distances built below have it, the sliced Wasserstein distance and the square root of the maximum mean discrepancy. A rate such as "the share of samples off the data" is not a distance between distributions, but it still splits usefully in practice, as the bench shows. This split is the first experiment to run when samples look wrong. An interviewer who asks "is it the sampler or the model?" wants to hear it.

Every loss is the noise loss with a weighting

The chapter has trained networks to predict four different things.

• The noise $\boldsymbol\epsilon$ (§33.2).
• The clean data $\mathbf{x}_0$ (§33.4).
• The velocity $\mathbf{v} = \alpha_t\boldsymbol\epsilon - \sigma_t\mathbf{x}_0$ on a variance-preserving path, one with $\alpha_t^2 + \sigma_t^2 = 1$ (§33.4).
• The flow-matching velocity $\mathbf{u} = \boldsymbol\epsilon - \mathbf{x}_0$ on the linear path, $\alpha_t = 1 - t$ and $\sigma_t = t$ (§33.6).

They look like four methods. At a fixed noisy point they are one prediction written in four units. The identity $\mathbf{x}_t = \alpha_t\mathbf{x}_0 + \sigma_t\boldsymbol\epsilon$ ties them together, so a guess of any one of them gives a guess of the other three. An error in one is then an error in the noise times a factor that depends only on the noise level.

Here is the simplest case with numbers. A noise guess $\hat{\boldsymbol\epsilon}$ implies the data guess $\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sigma_t\hat{\boldsymbol\epsilon})/\alpha_t$, so the data error is the noise error times $\sigma_t/\alpha_t = e^{-\lambda_t/2}$. At $\lambda = 0$ the factor is 1, and the two errors are equal. At $\lambda = -4$ the factor is $e^{2} = 7.4$, so a noise guess that is off by 0.1 gives a data guess that is off by 0.74. Squaring gives the factor $e^{-\lambda}$ on squared errors.

So the loss values of two parameterizations can never be compared directly. A flow-matching loss of 4 and a noise loss of 1 can describe the same network.

The per-point factor is half the story. The other half is how often training visits each noise level. A loss draws $t$ from a density $p(t)$, and through the schedule that becomes a density over $\lambda$. The product of the factor and that density is the weighting function $w(\lambda)$ of the loss, the weight per unit of log-SNR that it puts on the noise error. §33.2 introduced it for the variational bound.

In plain words, a loss is described by two things, how much it charges for an error at each noise level and how often it visits that level. Only their product matters for what the loss minimizes. Kingma and Gao (2023) made this point and tabulated the weightings used in the literature. Their $w$ is twice ours for the same loss, because they put a factor $\tfrac12$ in front, and their table also drops constant factors. A constant factor changes neither the minimizer nor the training run, since the steps of the Adam optimizer (§0.6) do not change when the loss is rescaled, apart from its small stabilizing constant.

The table collects the losses of this chapter. The middle columns are what the training code does, and the last two are the weighting it implies. Two rows come from specific recipes. Stable Diffusion 3 draws $t$ from a logit-normal density, meaning that $\operatorname{logit} t = \log\frac{t}{1-t}$ is Gaussian, which concentrates training on middle noise levels (Esser et al. 2024). The design of Karras et al. (2022), called EDM and met in §33.4, draws the noise level from a log-normal density and weights the loss so that the network's target has unit scale at every level. The last column asks whether $w$ never rises as the data get cleaner, a property we use below.

One row deserves a second look. Velocity prediction with the cosine schedule and flow matching with uniform $t$ have the same weighting up to the constant $2/\pi$. As objectives they are the same, which Kingma and Gao also observed. Two things still differ between them. One is the parameterization, meaning what the network outputs and how its input is scaled. The other is the density from which training draws $\lambda$. The bench below shows that, for a small network, these differences are not small.

The cell computes every weighting from its training recipe alone, with a numerical derivative of $t(\lambda)$, and prints where each one puts its weight.

The numbers match the table. Noise prediction with the cosine schedule is symmetric in $\lambda$. It charges a noisy level and a clean one equally, $w(-4)/w(4) = 1$. The velocity and flow-matching losses charge the noisy level $e^{4} = 54.6$ times more. EDM sits between them at 7.3, and DDPM's linear schedule at 3.7.

Keep in mind that $w$ multiplies the noise error, and the noise error itself vanishes at high noise. There $\mathbf{x}_t$ is almost all noise, so the guess $\hat{\boldsymbol\epsilon} \approx \mathbf{x}_t$ is nearly right for free. A large weight at high noise does not mean a large share of the loss.

When a weighting is a true bound

The simple noise loss drops the weights of the variational bound, and §33.2 showed that it then no longer bounds the likelihood of the data. There is still a precise sense in which many weighted losses are likelihoods. If the weight never grows as the data get cleaner, the loss is a likelihood bound for the data after extra Gaussian noise of a random level is added. Kingma and Gao (2023) proved this.

Read the last column of the table with this theorem. The bound itself, velocity prediction with the cosine schedule, and flow matching with uniform $t$ are monotone, so each is a likelihood of noise-augmented data. Noise prediction with either schedule, EDM, and the logit-normal density of Stable Diffusion 3 are not. Kingma and Gao then trained models with new monotone weightings. On $64\times64$ ImageNet, $w(\lambda) = \operatorname{sigmoid}(-\lambda + 2)$ beat the usual noise loss. A monotone version of EDM's weighting reached state-of-the-art FID at $128$, $256$, and $512$ pixels among models sampled without guidance. FID, the Fréchet Inception distance, compares feature statistics of generated and real images, and lower is better. That is an empirical result. Monotonicity makes an objective interpretable. It does not guarantee better samples.

import numpy as np

def weighting(lam, kind):
    return np.ones_like(np.asarray(lam, float))      # placeholder: weight 1 per unit of log-SNR

Measuring samples when the truth is known

A 2D lab has a luxury that image models never have, the true distribution. We can ask directly whether samples match it, with distances between distributions and with diagnostics that exist only because we know where the modes are. Image models are measured in the evaluation of image generators, with FID and with precision and recall for generative models. Those metrics are the same ideas applied to features of images.

Maximum mean discrepancy

The idea fits in one sentence. If $P$ and $Q$ are the same distribution, points drawn from $P$ look as similar to each other, on average, as they look to points drawn from $Q$, and the maximum mean discrepancy (MMD) measures how far that fails. "Similar" is measured by a kernel, a similarity score that is 1 for identical points and falls toward 0 as points move apart. It is the same discrepancy that the Wasserstein autoencoder of §32.4 uses to pull a cloud of codes toward its prior, and that section defines it and proves its estimate unbiased. Here is the short version, starting from two single points.

In general, take the Gaussian kernel $k(\mathbf{x}, \mathbf{y}) = \exp\big(-\lVert\mathbf{x} - \mathbf{y}\rVert^2/(2\ell^2)\big)$ with length scale $\ell > 0$. The squared discrepancy is

with $\mathbf{x}, \mathbf{x}' \sim P$ and $\mathbf{y}, \mathbf{y}' \sim Q$, all independent. For the Gaussian kernel it is zero if and only if $P = Q$, and its square root is a metric (Gretton et al. 2012), which is what the error split above needs. In plain words, blur both densities with a Gaussian of standard deviation $\ell/\sqrt2$, and measure the squared distance $\int(\tilde p - \tilde q)^2$ between the blurred densities $\tilde p$ and $\tilde q$. MMD$^2$ equals that up to a constant factor, because two such blurs in a row make the kernel.

From samples $\mathbf{x}_1, \dots, \mathbf{x}_n \sim P$ and $\mathbf{y}_1, \dots, \mathbf{y}_m \sim Q$ we use the unbiased estimate of §32.4,

whose expectation is exactly $\operatorname{MMD}^2(P, Q)$, because every pair it averages is a pair of independent draws. The diagonal terms $k(\mathbf{x}_i, \mathbf{x}_i) = 1$ are left out because a point is not independent of itself. Keeping them adds a positive bias even when $P = Q$. Because the unbiased estimate averages to zero when $P = Q$, it is often negative then. That is correct, not a bug.

The real choice is the length scale $\ell$. Much smaller than the spacing of the data, the kernel sees only near-duplicates. Much larger than the data, every distribution looks alike. In practice, pick $\ell$ near the size of the structure you care about, and report the score of exact data beside the model's.

Sliced Wasserstein distance

In one dimension, the cheapest way to move one sample onto another is to sort both and pair them in order.

So take two samples of the same size, sorted as $x_{(1)} \le \dots \le x_{(n)}$ and $y_{(1)} \le \dots \le y_{(n)}$. Their empirical 2-Wasserstein distance $W_2$, the root mean squared distance of the cheapest pairing, satisfies

§33.6 met $W_2$ in higher dimensions, where finding the cheapest pairing is an assignment problem. The sliced version avoids that problem by looking at many one-dimensional shadows of the data.

It is a metric, it costs one sort per direction, and it is measured in the units of the data. Its weakness is statistical. Two finite samples of the same distribution sit at a positive sliced distance from each other. That floor falls slowly with sample size, especially for data with empty space between modes, where sorted samples disagree most.

Diagnostics that need the truth

Because we know the eight modes, three numbers answer specific questions about the samples.

Even exact samples do not score perfectly. The distance $r$ of a sample of a 2D round Gaussian from its mean satisfies $r^2/s^2 \sim \chi^2_2$, which is exponential with mean 2. So an exact sample lands beyond $ks$ with probability $\mathbb{P}(r > ks) = e^{-k^2/2}$, which is 1.1 percent at $k = 3$. Sampling noise alone also leaves a small mode-weight error. So every metric needs its exact-data value printed beside it.

A likelihood is the remaining tool. A flow or diffusion model has an exact log-likelihood through the probability flow ODE and the instantaneous change of variables (§33.6). Held-out negative log-likelihood punishes missing modes heavily, because a held-out point in a dropped mode gets almost zero density. It is costly to compute, and §33.2 showed that it is a poor guide to how samples look. The reverse quantity, the average true log-density of the samples, measures fidelity only. A model that puts every sample on the densest point maximizes it, so it rewards collapse.

The cells build the metrics and apply them to six samples whose defects we chose. Each has 600 points and is compared with 600 exact reference points.

import numpy as np

def mmd2_unbiased(X, Y, bandwidth):
    raise NotImplementedError

def sliced_w2(X, Y, n_proj, rng):
    raise NotImplementedError

import numpy as np

def mode_metrics(samples, means, weights, std, k=3.0, min_frac=0.01):
    raise NotImplementedError

The sampler alone, measured with an exact denoiser

Now we measure the first term of the error split, the sampler's own error. One change of coordinates makes it easy to reason about. Divide the noisy point by its signal scale,

In these coordinates every Gaussian path looks the same, clean data plus noise of standard deviation $\sigma$. For example, at $\lambda = 0$ every path has $\sigma = 1$, and at $\lambda = -10$ every path has $\sigma = e^{5} = 148$. Only the speed at which $\sigma$ changes with $t$ differs from path to path. §33.4 used these coordinates to show that the samplers of all paths become one sampler.

In plain words, with a perfect network, "the DDIM sampler of a DDPM model" and "the Euler sampler of a flow-matching model" are the same algorithm. Both are Euler's method on $\mathrm{d}\mathbf{y}/\mathrm{d}\sigma = \hat{\boldsymbol\epsilon}$ (§33.6 proves the flow-matching side). The only thing a method's name can change is where the steps go.

And the steps do go to different places. Uniform steps in $t$ visit different noise levels on different paths. With $N$ steps, the last nonzero level is $\tan\!\big(\frac{\pi}{2}\cdot\frac{t_{\max}}{N}\big) \approx 1.57/N$ on the cosine path. On the linear path it is $\frac{t_{\max}/N}{1 - t_{\max}/N} \approx 1/N$. So the linear grid takes its final, most delicate step from a smaller noise level.

The stochastic sampler has the same property. DDPM's ancestral step, which adds fresh noise at every step, adds noise with variance $\sigma_{(i+1)}^2\big(1 - \sigma_{(i+1)}^2/\sigma_{(i)}^2\big)$ in $\mathbf{y}$ units. That is the posterior variance $\tilde\beta$ of §33.2 for a jump between two levels, divided by the new level's $\alpha^2$, and it too depends only on the levels.

Each entry is the off-manifold rate followed by the mode-weight error. With a perfect network the sampler is the whole story, and at small budgets it is a large one.

• At 4 calls, 43 to 56 percent of all samples are off the ring.
• At 8 calls the linear grid puts 4.7 percent off, against 10.2 percent for the cosine grid. The torch bench below, with other random draws, finds 5.2 and 10.6.
• From 16 calls on, both are close to the floor of exact data, 1.1 percent off and a mode-weight error of 0.02 from sampling noise alone.

The mode weights are right at every budget. The coarse structure is decided early, and the oracle decides it correctly. The stochastic sampler is no better at 4 or 8 calls, since its fresh noise has too few steps to be corrected. At 64 calls on the cosine grid it even comes in below the floor (0.000). That is a sign that its modes are too narrow, the under-dispersion of $\tilde\beta$ that §33.2 measured. A score better than exact data is a defect too.

The bench with trained networks

Now the second term, the network's error. The torch block trains one small network three ways on the ring, with the same architecture, optimizer, batch size, number of steps, and seed.

• The noise predictor uses the cosine variance-preserving path.
• The velocity predictor uses the same path.
• The flow-matching model uses the linear path.

Each draws $t\sim\mathcal{U}(0,1)$, as the original recipes do. The block then samples every model, and the oracle, with deterministic DDIM written in $\mathbf{y}$ coordinates. It does so on both grids and at 4, 8, 16, and 64 calls, so every factor can be changed while the others are held fixed. Each model's output is turned into a noise estimate with the conversions above, $\hat{\boldsymbol\epsilon} = \sigma_t\mathbf{x}_t + \alpha_t\hat{\mathbf{v}}$ on the variance-preserving path and $\hat{\boldsymbol\epsilon} = \mathbf{x}_t + (1-t)\hat{\mathbf{u}}$ on the linear path.

Each cell of the printed table holds the off-manifold rate, the mode-weight error, and MMD with $\ell = 0.1$. One seed is an anecdote, so we ran the block with five seeds, 0 to 4, and report each mean with a 95 percent interval.

Why 2.776 and not 1.96? The confidence intervals of §0.11 use the normal quantile 1.96, which treats the standard error as known. With five seeds it is estimated from five numbers, which adds uncertainty. For normally distributed results, the standardized mean $(\bar x - \mu)/(s/\sqrt5)$ then follows Student's t distribution with $5 - 1 = 4$ degrees of freedom, whose 0.975 quantile is 2.776. An interval built with 1.96 would be about 30 percent too narrow.

Read the table with the error split in mind.

At 4 calls the sampler dominates. The oracle itself puts 40 percent of its samples off the ring, and the flow-matching model matches it. A network's errors can even cancel part of the sampler's. The noise predictor puts only 32 percent off the ring at 4 calls, fewer than the oracle, while its mode weights are the worst in the table. One metric improved because two errors partly cancelled. That is why a bench never reports one metric.

From 16 calls on, the network dominates. The oracle reaches the floor. The flow-matching model stays at 10.2 percent off the ring at 64 calls, and the noise predictor at 14.2 percent. More steps cannot remove network error, so the curves flatten at different heights.

The noise predictor gets the mode weights wrong. Its mode-weight error is 0.178 at 8 calls, against 0.033 for flow matching. The error split locates the cause. The first DDIM step starts at $\lambda = -10$, where $\alpha_t = 0.0067$. It turns the noise estimate into a data estimate with $\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sigma_t\hat{\boldsymbol\epsilon})/\alpha_t$, which multiplies the network's error by $\sigma_t/\alpha_t \approx 150$. That first estimate decides which mode each sample heads for. In our seed-0 run on the cosine grid, replacing only that first of 8 calls by the oracle lowered the mode-weight error from 0.225 to 0.031. This is the failure at low signal-to-noise that §33.1 predicted. Velocity and data prediction were designed to avoid it (§33.4).

The velocity predictor gets the mode weights right and the modes wrong. Its samples are blurred, 46.9 percent off the ring even at 64 calls. The error sits at low noise. At 8 calls on the cosine grid, replacing only its last call by the oracle (the step from $\sigma = 0.20$ to 0) brought it from 61 to 13 percent off the ring, close to the oracle's 10. Its weighting is the same as flow matching's up to a constant, so the objective cannot explain the gap. The box below has the measurements and what they do and do not say.

So is flow matching better? Comparing the two rows of intervals throws information away. Every seed trains all three models, samples them from the same starting noise, and scores them against the same reference sample. So the right statistic is the per-seed difference, the paired comparison of §0.11. The variance of a difference is the sum of the two variances minus twice their covariance, so whatever a seed does to both models cancels. The five differences are independent across seeds, which together with rough normality is all the t interval needs.

At 8 and 16 calls both paired intervals exclude 0. So on this bench, at those budgets, flow matching beats noise prediction in fidelity and in mode weights. At 64 calls both intervals include 0, and five seeds cannot separate the two. At 4 calls the mode-weight gap is significant and the off-manifold gap is not, because of the cancellation described above.

The error split also says why. It is not the grid, since both models are sampled on the same grid, and the oracle on that grid is at 5 percent at 8 calls, far below both. It is network error, and for the noise predictor much of it enters through a single call, where the parameterization amplifies it. Neither reason is "straight paths" in the sense of §33.6. When noise and data are paired independently, all three models aim at the same probability flow, and its paths curve, whatever path each training pair follows. That is what a careful answer to the colleague sounds like.

The table below collects the design choices the bench separated, and which of them can still be changed after training. The last row is the exponential moving average (EMA) of the weights, which most models use for sampling instead of the raw weights.

The widget holds every run of the bench. Pick a metric, a time grid, and the mean over seeds or one seed, and read each method's curve against the network calls. Click a column to choose the number of calls, and a method to see its samples below the plot.

A perfect model would copy its training set

A good-looking sample can still be a bad sample, if it is a copy of a training image. The optimum of the training loss shows that this is a real risk. Train a denoiser perfectly on a finite training set, and it learns to reproduce exactly that set.

In plain words, the loss is minimized by a denoiser that, at low noise, snaps every point to its nearest training example. Real models generalize because they cannot fit that function. A network smooth enough to interpolate between training points learns something closer to the score of the underlying distribution. Kadkhodaie et al. (2024) trained two denoisers on non-overlapping subsets of a large image dataset and found that they learn nearly the same function, so their samples are new images. On small training sets the same networks memorize.

How do we detect a copy? A sample's nearest training point is not enough, since in dense regions every sample is close to something. Carlini et al. (2023) compared a generated image's nearest distance with its distances to the next-nearest training images. We use the simplest version of that idea.

The test is relative, so it adapts to how dense the training set is near the sample. It also has a clean baseline. Draw a sample from the data's own density, and condition on $d_2$. If the density is roughly uniform nearby, the nearest training point is uniform in the ball of radius $d_2$. So $\mathbb{P}(d_1 < \tau d_2) = \tau^d$, which is $1/9 = 0.111$ in two dimensions. A model whose flagged rate is well above $\tau^d$ is copying. Run the test against a deduplicated training set. A point stored twice is its own second neighbor, so $d_1 = d_2$, and a copy of it would never be flagged.

The cell samples twice with the same sampler. Once it uses the exact optimum for a training set of 64 points, and once the exact denoiser of the true ring. It uses the step spacing of Karras et al. (2022), which crowds the steps toward low noise and ends at the noise level 0.002.

The sampler with the exact empirical optimum flags 0.998 of its samples, and their median distance to the nearest training point is zero to machine precision. The true-ring oracle and fresh data both sit at about 0.10, close to the baseline of 0.111.

The last noise level of the grid matters here, more than the number of steps. EDM's grid ends at 0.002, far below the spacing between training points, so the final step snaps each sample exactly onto one of them. Eight uniform steps on the linear path end at 0.14 instead. The last step then averages several neighboring training points, and only 0.13 of the perfect copier's samples are flagged, close to the baseline. That is one reason memorization is easy to miss.

In practice, run a nearest-neighbor check against the training set before reporting any sample metric, and deduplicate the training data. An interviewer will ask why a perfect loss would be a disaster here. The theorem is the answer.

import numpy as np

def memorization_ratio(samples, train, tau=1/3):
    raise NotImplementedError

Debugging a diffusion model

A diffusion model whose loss goes down can still sample garbage. The loss is an average over noise levels, which can hide a level the network never learned. A sampler chains a hundred small errors into one large one. The list below gives each check and the failure it catches.

On item 7, §33.2 set the averaging horizon $1/(1-m)$ by hand, where $m$ is the EMA's decay. Karras et al. (2024) made it a choice after training. They store a few snapshots of averages during one run and reconstruct the average for any horizon afterwards. That let them tune the horizon precisely, and it showed that the best horizon interacts with architecture, training length, and guidance. Their redesigned network also keeps the magnitudes of activations and weights under control throughout training. Together these changes improved the record FID on $512\times512$ ImageNet from 2.41 to 1.81.

import numpy as np

def fixed_ddpm_sample(eps_model, n, rng, beta, alpha, sigma):
    return buggy_ddpm_sample(eps_model, n, rng, beta, alpha, sigma)     # find and fix the three bugs

Masked diffusion brings the idea to text

Text is a sequence of tokens, and a token has no "slightly noisy" version. Adding a Gaussian to a token ID is meaningless. The discrete replacement for adding noise is hiding tokens. Mask a random fraction of the tokens, train a network to fill them in, and generate by starting from an all-mask sequence and filling in a few tokens per step.

Formally, discrete diffusion replaces the Gaussian noise with a random process on the vocabulary. Austin et al. (2021) studied several, including replacement by a random token and moves to nearby tokens in embedding space. One stood out, the absorbing process, in which a token can only turn into a special mask token $\mathtt{m}$ and then stays masked. Its forward process is the masking used to train BERT, the encoder of §7.1, which learns to fill in tokens hidden at a fixed rate of 15 percent (§8.1 sets this masked language modeling objective beside next-token prediction). Masked diffusion runs that masking at every rate from 0 to 100 percent.

Here is a small example first. Take the four-token sentence "the cat sat down" at a time where each token survives with probability $\kappa_t = 0.75$. On average one token is hidden, giving, say, "the cat $\mathtt{m}$ down". At a later time with $\kappa_t = 0.25$, on average three are hidden, as in "$\mathtt{m}$ cat $\mathtt{m}$ $\mathtt{m}$".

Papers on masked diffusion write $\alpha_t$ for $\kappa_t$. We rename it because $\alpha_t$ is the signal scale in this chapter.

The reverse step needs one fact. Fix $s < t$ and a position that is masked at time $t$. Given $x_0^i$, the probability that it was still clean at time $s$ is

and if it was clean, it equals $x_0^i$. For example, with $\kappa_s = 0.8$ and $\kappa_t = 0.5$, a token masked at $t$ was clean at $s$ with probability $0.3/0.5 = 0.6$. So

and an unmasked position at time $t$ is unchanged at time $s$. The model's reverse step is the same formula with $x_0^i$ replaced by the denoiser's guess. It gives probability $\frac{\kappa_s - \kappa_t}{1 - \kappa_t}\,p_\theta(x^i = v\mid\mathbf{x}_t)$ to each real token $v$ and $\frac{1 - \kappa_s}{1 - \kappa_t}$ to $\mathtt{m}$. Sahoo et al. (2024) call this choice, zero mass on the mask and visible tokens carried over, the SUBS parameterization.

With that reverse step, the variational bound of §33.2 becomes a weighted fill-in-the-blank loss.

Two papers found this independently, Sahoo et al. (2024), whose model is called MDLM, and Shi et al. (2024), whose model is called MD4.

In plain words, pick a masking rate $t$ at random, mask that fraction of the tokens, ask the network to fill in the masked ones, and weight the cross-entropy by $1/t$. It is BERT's loss with two changes, a random rate instead of 15 percent, and the $1/t$ factor. The factor is what makes it a likelihood bound. At rate $t$ about $nt$ tokens are masked, so $\frac1t\sum_{\text{masked}}$ is, on average, the per-token loss times $n$, and every rate contributes on the same scale.

With the exact denoiser the bound is tight, and that gives masked diffusion a second reading. It is the same as generating the tokens one at a time in a random order.

The link to orders was made by Hoogeboom et al. (2022) and, for the absorbing process, by Ou et al. (2025). It also suggests why masked diffusion behaves differently from left-to-right models when data are scarce, the setting of §9.4. Prabhudesai et al. (2025) found that masked diffusion models extract far more from repeated data than autoregressive models. The theorem makes this plausible. Each pass over a sequence trains on a new random order, a form of augmentation that left-to-right training never gets.

Generating in parallel, and what it costs

To sample, start from $n$ masks and run the reverse chain. At each step the network predicts every masked position at once, and the step reveals some of them, each drawn from its own predicted distribution. If a step reveals one token, the chain is an any-order autoregressive model and samples exactly from $p_\theta$. If it reveals $k > 1$ tokens, it draws them independently given the current context. It then samples from the product of their marginals, not from their joint distribution. With few steps the model can produce combinations that no training sequence contains.

A two-token example shows the problem. Suppose the only valid pairs are "New York" and "Los Angeles", each with probability one half. Each position on its own is a fair coin between its two words. Revealing both at once draws two independent coins, so half the time it writes "New Angeles" or "Los York".

The cells make that concrete with a grammar of 18 five-token sentences of the form "the ADJ ANIMAL VERB PLACE". The verb depends on the animal, and the place depends on the verb. The denoiser is exact, the posterior over the 18 sentences given the visible tokens, so every error is the sampler's. The first cell checks the theorem. The second generates with 1 to 5 network calls, revealing positions in a random order or most confident first.

The bound equals $-\log p(\mathbf{x}_0)$ to four decimals, as the theorem says for an exact denoiser. Any excess from the $10^{-6}$ smoothing is below the fourth decimal. Generation shows the price of parallelism.

• With one call, all five tokens are drawn from their marginals at once, so the order is irrelevant. The valid rate is then the product-of-marginals value $3/32 = 0.094$. The cell's 0.092 and 0.094 are two Monte Carlo estimates of it, each from 20,000 sentences.
• In random order each extra call helps, and five calls, one token each, are always valid.
• Revealing the most confident positions first is much worse at two calls, 0.094 against 0.631.

The confident order fails because "the" and the adjective are the most confident tokens and also the ones that say nothing about the rest. The first call is spent on them. The three dependent tokens are then drawn together in the second call, each from its own marginal, exactly as one call does. Four calls differ from three only in revealing the adjective on its own instead of beside the place, so they are no better either. Confidence is not independence. What a step can safely reveal together are tokens that are nearly independent given what is already visible.

The widget runs the same grammar. Choose the number of network calls and the order, press Sample, and watch which positions each call reveals and which sentences come out invalid. The large number is the exact valid rate, summed over every order and every draw, so it differs slightly from the cell's estimates.

import numpy as np

def masked_diffusion_loss(logits_fn, x, mask_id, rng):
    raise NotImplementedError

Revealing only near-certain tokens

The grammar showed that confidence is not independence. But confidence near 1 does force near independence. A practical rule follows. At each step, reveal every masked position whose top probability exceeds a threshold, instead of a fixed number of positions. Wu et al. (2025) use this rule in a system called Fast-dLLM, and the theorem below is why it is safe.

In the grammar, "the" has probability 1 and could join any call for free, while the animal, at 0.25, could not. A threshold of $1 - \varepsilon$ with $k\varepsilon$ small keeps the parallel step close to the joint. When no position is confident enough, the sampler falls back to one token per call. The Fast-dLLM paper also proves a sharper statement for greedy decoding, which picks the most likely tuple. The product's most likely tuple equals the joint's when $(k+1)\varepsilon \le 1$.

Diffusion language models at scale

LLaDA (Nie et al. 2025) scaled the recipe. It is a Transformer with no causal mask and 8B parameters. It was pre-trained from scratch on 2.3 trillion tokens with exactly the loss of the theorem, at a cost of 0.13 million H800 GPU-hours. It was then fine-tuned on 4.5 million instruction pairs by masking only the response. It was competitive with LLaMA 3 8B on many benchmarks and better at a reversal task that left-to-right models find hard.

For sampling, LLaDA predicts all masked tokens at each step and then masks again the least confident fraction of its predictions, which it calls low-confidence remasking. It also supports semi-autoregressive blocks, generated left to right with diffusion inside each block. Its headline results use as many steps as generated tokens, one token per network call. That is the regime where the parallel-decoding error above vanishes.

Commercial systems trade some quality for parallelism. An independent benchmark measured Inception's Mercury Coder Mini and Small at 1,109 and 737 tokens per second on an H100 (Inception Labs 2025). Gemini Diffusion is an experimental Google DeepMind model. Its page reports 1,479 tokens per second of sampling. It scores slightly above Gemini 2.0 Flash-Lite on the LiveCodeBench v6 coding benchmark (30.9 against 28.5 percent), but well behind it on the GPQA Diamond science questions (40.4 against 56.5). ByteDance's Seed Diffusion Preview reports 2,146 tokens per second on H20 GPUs (Song et al. 2025). Each speed was measured on its own hardware and serving setup, so the numbers do not compare directly. Dream 7B (Ye et al. 2025) is an open model that starts from an autoregressive model's weights. That suggests the two kinds of model are closer than their samplers make them look.

The limits are real too.

• Fixed length. A diffusion LM generates a fixed-length canvas, so the length must be chosen or padded. LLaDA's instruction model pads with end-of-sequence tokens, which hurt its low-confidence sampler on some tasks.
• No exact cache. Attention is bidirectional, so the keys and values of every position change as tokens are revealed. The exact KV cache of §6.5, which stores past keys and values so each new token costs one position, does not apply to a plain masked diffusion LM. A plain sampler recomputes the whole sequence at every step. Fewer calls can still be slower if each call costs more.
• A strong rival. Autoregressive models have their own route to parallel decoding, speculative decoding, in which a cheap draft model proposes several tokens and the large model checks them in one pass, keeping its exact distribution.

Work since 2025 has narrowed both gaps. Block diffusion (Arriola et al. 2025), called BD3-LM, splits the sequence into fixed-size blocks. It generates the blocks left to right and runs masked diffusion inside each block, with attention causal across blocks. A finished block never changes, and later blocks cannot alter its keys and values. So it is cached exactly, as in an autoregressive model, and generation can run to any length. Block size 1 is an autoregressive model, and one block covering the whole sequence is the masked diffusion of this section. The block size interpolates between the two.

The state of the art in October 2026

The map changes monthly, so read it by kind of claim. Architectures and objectives of open models come from their papers and are reliable. Benchmark records are reliable for the benchmark and the date. Speeds and comparisons from vendors' pages are claims about their own setups. One thing is constant across all of them, and it is the subject of this chapter. Every one of these systems is trained to remove noise or follow a velocity at many noise levels, and every one is sampled by integrating what it learned.

Where it breaks

• Comparisons that fix the wrong thing. Equal steps instead of equal network calls, different time grids, one seed, or one metric. Each one changed a conclusion on our bench.
• Metrics with blind spots. Sliced distances have a high floor and miss wrong widths. MMD's verdict depends on its length scale, and it misses small off-manifold mass. Coverage counts miss wrong widths. Report several, with the score of exact data beside each.
• Toy results that do not transfer. A 2D ring is a test of algorithms, not of architectures. The velocity predictor's trouble on our bench says nothing about large models. The bench's value is that it can say why a method failed, not which method wins at scale.
• Memorization mistaken for quality. A model that copies its training set scores well on every distribution metric computed against the training data. Check nearest neighbors against the baseline of fresh data.
• Discrete diffusion's sampler. Parallel unmasking samples products of marginals. Quality falls when too many dependent tokens are revealed in one step, unless only near-certain tokens are revealed together. The bidirectional network also rules out the exact KV cache, unless generation is split into causally attending blocks.

Read more

• Kingma & Gao (2023). Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation. The weighting view of every diffusion loss, its Table 1 of weightings, the importance-sampling reading of the schedule, and the monotone-weighting theorem proved here.
• Karras, Aittala, Lehtinen, Hellsten, Aila & Laine (2024). Analyzing and Improving the Training Dynamics of Diffusion Models. EDM2, with magnitude-preserving layers and post-hoc EMA, the practical reference for training and averaging.
• Carlini, Hayes, Nasr, Jagielski, Sehwag, Tramèr, Balle, Ippolito & Wallace (2023). Extracting Training Data from Diffusion Models. How memorization was measured and exploited at scale, and what made models memorize.
• Kadkhodaie, Guth, Simoncelli & Mallat (2024). Generalization in diffusion models arises from geometry-adaptive harmonic representations. The transition from memorization to generalization as the training set grows.
• Sahoo, Arriola, Schiff, Gokaslan, Marroquin, Chiu, Rush & Kuleshov (2024). Simple and Effective Masked Diffusion Language Models. The masked diffusion bound as weighted masked-LM losses, the SUBS parameterization, and the engineering that closed most of the gap to autoregressive perplexity.
• Shi, Han, Wang, Doucet & Titsias (2024). Simplified and Generalized Masked Diffusion for Discrete Data. MD4, the same continuous-time bound found independently as a weighted integral of cross-entropies, with masking schedules that depend on the token.
• Nie, Zhu, You, Zhang, Ou, Hu, Zhou, Lin, Wen & Li (2025). Large Language Diffusion Models. LLaDA, masked diffusion at 8B parameters, with its training and remasking algorithms written out.
• Arriola, Gokaslan, Chiu, Yang, Qi, Han, Sahoo & Kuleshov (2025). Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. BD3-LM, autoregressive over blocks and diffusion within them, with exact KV caching and generation of any length.
• Austin, Johnson, Ho, Tarlow & van den Berg (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. D3PM, the general discrete diffusion framework from which the absorbing chain emerged.
• Gretton, Borgwardt, Rasch, Schölkopf & Smola (2012). A Kernel Two-Sample Test. The theory of MMD, its unbiased estimator, and when it is a metric.

Watch next

▶

▶

▶