This section is about deciding whether one image or video generator is better than another. You make that decision every time you train one. It picks the checkpoint you ship and the guidance scale you set, and it decides which published results you believe. The standard numbers often disagree with people, and the reasons are specific enough to learn and to fix.

Two cases show the problem. The most widely used score is FID, the Fréchet Inception distance. It compares summary statistics of generated images with those of real ones, and lower means a closer match. The SDXL paper computed it for SDXL and for Stable Diffusion 1.5 and 2.1 on 10,000 captions from COCO, a dataset of photos with human-written captions (Podell et al. 2023, Figure 12). SDXL had the worst FID of the three. In the same paper, people were shown generations from four models and picked their favorite. They chose SDXL with its refiner 48.4 percent of the time and the SDXL base model 36.9 percent, against 7.9 and 6.7 percent for versions 1.5 and 2.1.

The second case is sharper. Muse is a text-to-image model that builds an image by filling in discrete tokens over a fixed number of refinement steps (§34.4). Jayasumana et al. (2024) made a deliberately worse version by stopping early, after 20 of its 24 base steps and 3 of its 8 super-resolution steps. The worse model had the better COCO FID, 18.42 against 21.40. Raters then compared pairs of images made from the same random seeds, one from each model. They preferred the full model's image 92.5 percent of the time.

So the most trusted number moved the wrong way twice. This section explains why, and what to measure instead. We start with what a generator must get right and why likelihood does not settle it. Then we build the Inception score and FID, prove that FID is a transport distance between two Gaussians, and show that its estimate is biased by an amount that depends on the model. Next come the metrics that each fix one flaw, for the features, for fidelity against diversity, for copies, for prompt adherence, and for video. We end with human studies, the reward models trained to imitate them, and how to report a number that someone else can trust.

What a good generator must get right

A generator can fail in five distinct ways, and a useful evaluation tells them apart. The rest of the section uses these five names.

• Fidelity. Individual samples look wrong, with blur, warped hands, or broken text.
• Diversity. Each sample looks fine, but together they cover only part of the data. The modes of a distribution are its peaks, the distinct kinds of image it contains. A generator that misses some of them is said to drop modes. Every face coming out the same age is a typical case.
• Adherence. For a model that takes a condition such as a prompt, the image does not match it. The count is wrong, or a color sits on the wrong object.
• Novelty. Samples are copies, or near-copies, of training images.
• Preference. Everything above is fine, and people still prefer another model's images for their look, composition, or style.

Cost is a sixth axis, measured in network evaluations or seconds per image. Every comparison should report it, but it is easy to measure, so this section does not return to it. When an interviewer asks how you would evaluate a generator, name these axes separately before you name any metric.

Why likelihood does not settle it

Many generative models are trained to make real images probable. The likelihood of an image is the probability density that the model assigns to it. On held-out test images it is reported as the average log-likelihood, $\frac1n\sum_i\log p(\mathbf{x}_i)$, in nats when the logarithm is natural, and higher is better. Autoregressive models compute it exactly, and diffusion models give a bound on it (§33.7). So why not rank generators by it?

Because in high dimension a good likelihood says almost nothing about how the samples look. Here is a concrete case. Take a good model and mix it with pure noise, so that 99 samples in 100 are noise. The mixture's samples are garbage, yet its log-likelihood barely moves. Theis et al. (2016) made this argument and credit it to van den Oord and Dambre.

How small is 4.61 nats? A $32\times32$ color image has $D = 3{,}072$ pixel values, and the log-likelihoods that different models give such images can differ by thousands of nats. A loss of 4.61 is invisible at that scale. Papers usually report likelihood in bits per dimension. Spread over 3,072 values, 4.61 nats is $4.61/(3{,}072\ln 2) \approx 0.002$ bits per dimension, below the precision of most tables. So a model whose samples are 99 percent noise can still rank near the top.

The reverse also happens. A model that outputs training images plus imperceptible noise has superb samples and a poor likelihood on held-out data.

Likelihood measures something real, how much probability a model puts on new data, which is what compression needs. It does not measure whether samples look right. So image generators are judged by sample-based metrics, which look at images the model actually produces.

The Inception score

The first widely used metric needs no real images at all (Salimans et al. 2016). It runs each generated image through an Inception network, a convolutional classifier trained on the 1,000 object classes of ImageNet, and reads off the predicted class probabilities $p(y\mid\mathbf{x})$. It rewards two things. Each image should look clearly like one thing, so its prediction should be peaked, like $(0.9, 0.05, 0.05)$ over three classes. And the generator should make all kinds of things, so the average prediction over many images should be spread out, like $(\tfrac13, \tfrac13, \tfrac13)$.

The score compares each prediction with the average prediction through the KL divergence $D_{\mathrm{KL}}(p\,\Vert\,q) = \sum_y p(y)\log\frac{p(y)}{q(y)}$, which is zero when two distributions agree and grows as they differ (§0.5).

Why does one KL term reward both properties at once? Because its average is a familiar quantity, the mutual information $I(y;\mathbf{x}) = H(y) - H(y\mid\mathbf{x})$. It measures how much seeing the image reduces your uncertainty about its class. Here $H(y)$ is the entropy of the marginal, which is large when many classes appear, and $H(y\mid\mathbf{x})$ is the average entropy of one prediction, which is small when each prediction is confident.

In plain words, the Inception score is the effective number of classes that the generator produces confidently, a number between 1 and 1,000.

The theorem also shows what the score cannot see. It never looks at real images, so it cannot tell whether the samples resemble your training data. A generator that outputs one perfect image per class gets the maximum score, $K_{\text{cls}}$, with no variety within any class. It knows only ImageNet's classes, so on faces or bedrooms it measures something unrelated. And it is fragile. Barratt and Sharma (2018) scored the 50,000 training images of CIFAR-10, a dataset of small $32\times32$ photos in 10 classes, with three public implementations of the Inception network, one of Inception-v2 and two of Inception-v3, whose ImageNet top-1 accuracies lie between 75.6 and 77.7 percent. They got 11.24, 9.74, and 10.85, gaps larger than many published improvements. Today the score survives mainly in tables for class-conditional ImageNet models, and interviewers still ask you to derive it.

import numpy as np

def inception_score(P, n_splits=10, eps=1e-12):
    raise NotImplementedError

The Fréchet distance between Gaussians

The Inception score never looks at real images, so the natural fix is to compare generated images with real ones. Heusel et al. (2017) did it in three steps. Run every image through the Inception network and keep the 2,048 numbers of its last pooling layer as the image's feature vector. Fit a Gaussian, described by a mean vector and a covariance matrix, to the features of the real images, and another to the features of the generated ones. Then measure the distance between the two Gaussians.

The distance they chose is the Fréchet distance, and it has a concrete meaning. Imagine moving the probability mass of one distribution until it becomes the other, and paying the squared distance that each bit of mass travels. The Fréchet distance between two Gaussians is the cheapest possible average cost. We first work it out in one dimension, where everything can be done by hand.

One dimension, by hand

A coupling of two distributions $P$ and $Q$ is a way to pair their samples. Formally, it is a joint distribution of a pair $(X, Y)$ with $X \sim P$ and $Y \sim Q$. Think of it as a transport plan that says which point of $P$ moves to which point of $Q$. The 2-Wasserstein distance is the cost of the cheapest plan, measured as the mean squared distance moved (§33.6 used it to pair noise with data),

Let $X \sim \mathcal{N}(\mu_1, \sigma_1^2)$ and $Y \sim \mathcal{N}(\mu_2, \sigma_2^2)$. For any coupling, expanding the square around the means gives

because the cross term $2(\mu_1 - \mu_2)\,\mathbb{E}[(X - \mu_1) - (Y - \mu_2)]$ is zero. Only the covariance depends on the coupling, so the cheapest plan makes it as large as possible. The Cauchy-Schwarz inequality caps it at $\operatorname{Cov}(X, Y) \le \sigma_1\sigma_2$. The monotone coupling $Y = \mu_2 + (\sigma_2/\sigma_1)(X - \mu_1)$ reaches the cap, since it has the right distribution for $Y$ and covariance exactly $\sigma_1\sigma_2$. So

In plain words, line up the two bell curves by their quantiles and move each point straight across. A wrong mean costs its squared error, and a wrong spread costs the squared error in standard deviation.

Any dimension

In $d$ dimensions the same expansion works with one change. The part that depends on the coupling becomes the trace of the cross-covariance, the $d\times d$ matrix of covariances between the coordinates of $X$ and those of $Y$. The question is how large that trace can be for two Gaussians with given covariances. The answer needs a matrix square root. A symmetric matrix $\mathbf{A}$ is positive semidefinite (PSD) when $\mathbf{u}^\top\mathbf{A}\mathbf{u} \ge 0$ for every vector $\mathbf{u}$. It has an eigendecomposition $\mathbf{A} = \mathbf{Q}\boldsymbol\Lambda\mathbf{Q}^\top$ with nonnegative eigenvalues, and $\mathbf{A}^{1/2} = \mathbf{Q}\boldsymbol\Lambda^{1/2}\mathbf{Q}^\top$ is its unique PSD square root (§0.10).

The proof below needs only the one-dimensional argument, a fact about block matrices, and the singular value decomposition. The formula was first proved by Dowson and Landau (1982) and by Olkin and Pukelsheim (1982), and the bound by Gelbrich (1990).

In plain words, FD is the cost of the cheapest transport plan that carries one Gaussian onto the other. Its square root is a true metric between Gaussians. Applied to other distributions, the formula sees only their means and covariances. So it is zero whenever those agree, even if the distributions differ, and by the Gelbrich bound it never exceeds their true $W_2^2$.

How do you compute the trace term in code? One way takes a general matrix square root of the non-symmetric product $\hat{\boldsymbol\Sigma}_r\hat{\boldsymbol\Sigma}_g$, but that is fragile. scipy.linalg.sqrtm can return complex numbers with tiny imaginary parts, especially when a covariance is singular, and the widely used pytorch-fid library checks them and throws them away. The symmetric form needs only symmetric eigendecompositions, whose eigenvalues are real by construction. Clip the tiny negative eigenvalues that rounding produces before taking square roots. The cost is $O(Nd^2)$ for the covariances and $O(d^3)$ for the eigendecompositions.

The symmetric form and the principal square root agree to every printed digit, and the naive product of square roots is off by 17 percent. With 40 samples of 64 features the covariances have rank 39, and sqrtm returns a complex matrix. Real FID code meets this case whenever a feature set has fewer images than feature dimensions.

import numpy as np

def fid_from_features(F_r, F_g):
    raise NotImplementedError

FID is biased, and the bias depends on the model

FID is computed from a finite sample of images, so it is an estimate of a true value, the FD between the two feature distributions. An estimator is biased when its average over many repeats differs from the true value. FID's bias is large, it points upward, and it differs from model to model.

Where does the bias come from? FID has a mean term and a covariance term, and the mean term is the easy half.

In plain words, two sample means never land exactly on each other, even when the true means agree. Their expected squared gap is the total variance divided by the sample size.

The covariance half has no closed form, but its sign is the same for every distribution.

So the bias always points up, and its size depends on the distributions themselves. That is why it differs from one model to the next.

How large is the bias? For Gaussian features, expanding the matrix square roots to second order gives it explicitly.

In plain words, the covariance part of the bias counts pairs of feature directions, weighted by their variances, and divides by $N$. With a flat spectrum it grows like $d^2/N$, so high-dimensional features need far more samples than the mean alone would suggest.

Two samples of one distribution score 4.44 at $N = 500$, 1.09 at 2,000, and 0.28 at 8,000. That is the $1/N$ decay the formula predicts (4.42, 1.10, 0.28). The general-spectrum formula agrees with its simulation to within about 2 percent, the noise of 20 repeats plus the neglected higher-order terms. Real Inception features have 2,048 dimensions with a decaying spectrum, so their bias is smaller than that of a flat spectrum with the same total variance. But it is never zero, and it differs from model to model.

The bias can reorder models

The bias depends on the generator's own covariance, so two models evaluated at the same $N$ can come out in the wrong order. In the next cell the reference statistics are exact and the true distances are known. Model A is 18 percent too spread out in every direction, with true FD 1.04. Model B is 20 percent too concentrated, with true FD 1.28. So A is the better model. But a concentrated generator has small sample covariances and therefore a small bias, and that flatters B.

At 250 and 500 samples B looks better, 2.24 against 2.47 and then 1.73 against 1.79. From 1,000 samples on, the order is right. This is not an artifact of Gaussians. Chong and Forsyth (2020) found the same reversals with real generative adversarial networks (GANs), even between two independently trained copies of one architecture. A GAN trains its generator against a second network, the discriminator, which tries to tell generated images from real ones (§32.6). Chong and Forsyth concluded that no single $N$ makes comparisons safe, because the bias at $N$ samples depends on the generator.

The extrapolation is the last line of the cell. FID is nearly linear in $1/N$, so Chong and Forsyth compute it at several sample sizes, fit a straight line in $1/N$, and report the intercept, $\operatorname{FID}_\infty$, the value at $1/N = 0$. Their protocol uses 15 sample sizes from 5,000 to 50,000. In the cell the intercepts are 1.040 and 1.278 against true values of 1.037 and 1.280. That is within 0.003 for both models, and much closer than the FIDs at 8,000 samples (1.07 and 1.31).

The widget shows the bias and the extrapolation together. Its generated features are Gaussian in $d$ dimensions, either from the real distribution itself or shifted by 0.5 along one axis and 20 percent narrower, and the reference statistics are exact. The blue curve is the mean FID of many runs at each $N$, with their range shaded around it. The dashed line is the true distance, drawn only when it is above zero, since a log axis has no zero. The dotted line is the straight-line fit in $1/N$ over the sample sizes in the green band, from $N/10$ to $N$. The dot in the last column, marked ∞, is its value $\operatorname{FID}_\infty$ at $1/N = 0$, and a triangle pointing down replaces it when the fit falls to zero or below.

import numpy as np

def fid_infinity(sample_features, mu_r, S_r, Ns, rng):
    raise NotImplementedError

In one dimension the whole bias has a closed form, because the expected value of a Gaussian sample's standard deviation is known exactly. That lets you compute, rather than simulate, how many samples a comparison needs before the bias stops reversing it. Bińkowski et al. used the same one-dimensional case to show the reversals analytically (their Appendix D.1).

import numpy as np

def expected_fd_1d(mu_p, s_p, mu_q, s_q, m, n):
    return (mu_p - mu_q) ** 2 + (s_p - s_q) ** 2       # placeholder, the true distance, which ignores the bias

def first_correct_m(p1, p2, q, n, m_max=10**6):
    raise NotImplementedError

What FID's features see, and what they miss

So far we treated the feature map $\phi$ as given. It matters at least as much as the distance, and for modern models the evidence against Inception features is strong. There are four findings.

The features are not Gaussian. Pool3 activations pass through a ReLU, which sets negative values to zero, and then an average over image positions. So they are never negative, and Bińkowski et al. report that about 2 percent of them are exactly zero. FID uses only means and covariances, so it cannot see any difference beyond those two moments.

The features belong to an ImageNet classifier. Kynkäänniemi et al. (2023) showed that FID looks mostly at whichever ImageNet objects the Inception network thinks it sees. They tested this on FFHQ, a dataset of face photos, with maps of the pixels that FID is most sensitive to. Those maps highlight seat belts and suits more than the faces. Then they kept a face generator fixed and only chose which of its outputs to keep. The generator was StyleGAN2, a strong GAN for faces. First they kept a subset of its outputs chosen so that the subset's histogram of top-1 ImageNet predictions matched the real set's. That alone lowered FID by about 10 percent. Weighting the samples to minimize FID directly took FID from 5.30 to 1.78. That is a 66 percent improvement with no visible change in the images. The same distance computed on CLIP features fell only 4 percent, from 2.76 to 2.64. Weighting the samples only to match which top ImageNet classes each image shows brought FID most of the way down as well, so the drop is mostly about ImageNet classes.

The features undervalue diffusion models. Stein et al. (2023) ran what was then the largest human study of generative models, with more than 1,000 paid participants and 207,000 responses about images from 41 models. They measured how often people mistake a generated image for a real one. Diffusion models fooled people more often than GANs did on every dataset but FFHQ, where the two were on par, yet metrics built on Inception features ranked them below GANs. None of the 17 metrics they tested correlated strongly with the human judgments. Replacing Inception-v3 with DINOv2 ViT-L/14, a self-supervised image encoder (§28.4), restored the agreement between the Fréchet distance and people. The Fréchet distance in those features is now called $\operatorname{FD}_{\text{DINOv2}}$.

The features see pixels you cannot. Parmar et al. (2022) showed that FID reacts to processing that people cannot see. Saving FFHQ images as JPEG at quality 75, the default of the PIL image library, gives an FID of 20.96 against the same images saved as PNG. A strong generator scores about 3, and even quality 90 gives 6.08. A StyleGAN2 trained on LSUN churches, a dataset of church photos stored as JPEG, scored 4.00 with PNG outputs and 3.48 after its outputs were JPEG-compressed at quality 87. The improvement was bought by post-processing.

Resizing matters as much. The TensorFlow and PyTorch FID code resized images with a bilinear filter of fixed width. When such a filter shrinks an image a lot, it aliases, folding fine detail into false coarse patterns. PIL's bicubic filter widens with the scale factor and does not. The two give different FIDs on identical images. Parmar et al. released the clean-FID library, which resizes with a correct filter.

These four findings are the answer to a favorite interview question, how a model can improve FID without making better images.

Kernel distances, KID and CMMD

Two of FID's problems, the Gaussian assumption and the bias, go away if we compare distributions with a kernel instead. A kernel $k(\mathbf{a}, \mathbf{b})$ is a similarity score between two feature vectors. The squared maximum mean discrepancy (MMD) uses it to compare two distributions. If $P$ and $Q$ are the same, a point of $P$ is as similar, on average, to another point of $P$ as it is to a point of $Q$. MMD measures how far that fails (§33.7),

with $\mathbf{x}, \mathbf{x}' \sim P$ and $\mathbf{y}, \mathbf{y}' \sim Q$, all independent. That section proved that averaging over distinct pairs, leaving out each point's similarity with itself, gives an estimate with no bias at all. Two image metrics are built on it.

Because it is unbiased, KID averages to its true value at any sample size. It can come out negative when the true value is near zero, which is correct, and it must never be clipped. Its spread over subsets is an honest measure of uncertainty. But what does the cubic kernel actually compare?

So KID sees one moment more than FID, and no further. The Gaussian kernel $k(\mathbf{a}, \mathbf{b}) = \exp\big(-\lVert\mathbf{a} - \mathbf{b}\rVert^2/(2\ell^2)\big)$ of §33.7, with length scale $\ell$, is different. It is characteristic, meaning that its MMD is zero only when the two distributions are equal. The cell tests this on two very different 1D distributions that share their mean (0), variance (1), and third moment (0). One is uniform on $[-\sqrt3, \sqrt3]$, and the other is a coin that lands on $-1$ or $+1$.

FD and KID both see nothing, with values at the level of the sampling noise between two samples of the same uniform. The Gaussian kernel with length scale 0.5 separates the two distributions by more than ten times its noise. That is the case for a characteristic kernel.

CMMD takes that route at scale (Jayasumana et al. 2024). It embeds images with the CLIP ViT-L/14 image encoder at 336 pixels (§30.3), uses the Gaussian kernel with length scale $\ell = 10$ (the paper calls it the bandwidth $\sigma$), takes the unbiased $\operatorname{MMD}^2$, and multiplies it by 1,000 for readability. It assumes no Gaussian shape and has no bias in $N$, and in their experiments it settled with far fewer samples than FID. On the Muse case of the opening it agreed with the raters, scoring 0.721 for the full model against 0.951 for the truncated one, while both FID and $\operatorname{FID}_\infty$ preferred the truncated model. Its features bring CLIP's own blind spots with them (§30.6), so it is a better default, not a final answer.

import numpy as np

def kid(F_r, F_g):
    raise NotImplementedError

def kid_subsets(F_r, F_g, n_subsets, subset_size, rng):
    raise NotImplementedError

Precision and recall for generative models

Every distance so far folds fidelity and diversity into one number. The 1D example showed what that costs, since a too-wide and a too-narrow generator scored the same. Sajjadi et al. (2018) proposed measuring the two separately, and Kynkäänniemi et al. (2019) gave the version in common use. It estimates where each distribution lives by putting a ball around every sample.

Here is the idea before the symbols. Around each real image's feature vector, draw a ball that reaches its third-nearest real neighbor. The union of these balls estimates the region where real images live. A generated image counts as realistic if it lands inside some real ball. Do the same with balls around the generated images to estimate where the generator's images live.

The names come from classification (§0.11), where precision is the share of predicted positives that are right and recall is the share of true positives that are found. Here precision is the share of generated images that land where real images live, so it measures fidelity. Recall is the share of real images that the generator could have produced, so it measures diversity.

The failures now separate cleanly. Dropping modes lowers recall and leaves precision alone. Off-manifold samples, such as artifacts or haze, lower precision. Guidance raises precision and lowers recall. On class-conditional ImageNet at $256\times256$, adding classifier guidance at scale 10 took precision from 0.69 to 0.88 and recall from 0.63 to 0.32 (Dhariwal and Nichol 2021, discussed in §33.5).

The balls have a weakness. A single real outlier far from everything gets a huge ball, because its $k$-th neighbor is far away, and every generated point inside that ball counts as realistic. Naeem et al. (2020) fixed this in two ways. They count how many real balls contain a sample, instead of asking whether any does. And they use only real balls, which are computed once per dataset.

Unlike precision and recall, both have exact expected values when the generator is perfect. So you know what "good" looks like before you measure.

Now put all four numbers to work. The cell scores five generators against real data made of eight Gaussian modes on a ring, and it prints the Fréchet distance beside them. Before you run it, guess which of precision and recall drops for "3 modes dropped", and to roughly what value. Guess which number suffers for "too narrow", the generator that looks like the effect of strong guidance. And ask whether the FD column can tell you which failure it saw.

Read the table by columns. Dropping three of eight modes cuts recall and coverage to about $5/8$ and leaves precision and density where a perfect generator puts them. Narrow modes keep every sample realistic (precision 0.99) and lose the spread (recall 0.53), the guidance trade-off in miniature. Wide modes and haze cost precision and density.

The exact generator's precision and recall are 0.96, not 1, and nothing tells you in advance what value to expect. Its density of 1.00 and coverage of 0.96 match the theorem's 1 and 0.969. Now look at the FD column. It is the same for the exact, narrow, and wide generators (0.008 to 0.009), because the spread within one mode is a tiny part of the whole mixture's covariance. It rises a little for haze and jumps only when dropped modes move the mean and reshape the covariance. The first two moments of a mixture say almost nothing about its modes.

The widget runs the same ring in your browser. Blue points are real and pink points are generated. A filled pink point lies inside some real ball. A hollow pink point lies outside every real ball and costs precision, and a hollow blue point lies outside every generated ball and costs recall.

import numpy as np

def precision_recall(F_r, F_g, k=3):
    raise NotImplementedError

Memorization, measured at scale

A sample can be realistic and diverse and still be a copy of a training image. §33.7 showed why this is a real risk. The denoising loss on a finite training set has an exact minimizer, and that minimizer generates only training images. Real models generalize because they cannot fit it exactly. That section also built a copy detector for 2D data, which flags a sample whose nearest training point is much closer than its second-nearest. Two things change at image scale. Nearest neighbors in pixel space are dominated by backgrounds. And the training set has billions of images, so you cannot compare every sample with every training image.

Carlini et al. (2023) designed the extraction attack that is now the reference measurement. A training image $\mathbf{x}$ is $(\ell, \delta)$-extracted if an attacker who has never seen it can produce a sample $\hat{\mathbf{x}}$ with $\ell(\mathbf{x}, \hat{\mathbf{x}}) \le \delta$. Here $\ell$ is the root mean squared pixel difference, with pixels in $[0, 1]$.

Their search on Stable Diffusion v1.4 rests on one observation. A model that generalizes makes different images from different random seeds. So if ten generations from one caption are near-identical, the model is probably replaying an image it memorized. The pipeline has three steps.

1. Choose the 350,000 captions whose training images were most duplicated in the training set.
2. Generate 500 images per caption, 175 million in all.
3. Flag a caption when at least 10 of its 500 generations are near-identical to one another. To compare two images, cut each into 16 tiles and take the largest distance between corresponding tiles. Plain pixel distance makes many generations look close just because they share a gray background.

Does the image match the prompt

For a text-to-image model the most important question is often adherence, not realism. Does the image show what was asked? The first tool was CLIP, a pair of encoders for images and text, trained so that a photo and its caption get similar unit vectors (§30.1).

CLIP score inherits CLIP's strengths and its blind spots. Contrastive training rarely rewards word order, counting, or spatial relations (§30.6). So "a red cube on a blue sphere" and "a blue cube on a red sphere" get nearly the same score. Binding failures, where an attribute lands on the wrong object (§34.3), pass unnoticed. The field's answer is to ask specific questions about the image and let a vision model answer them. A visual question answering (VQA) model is a network that answers questions about an image. Four benchmarks show the range.

• TIFA (Hu et al. 2023) uses a language model to turn each prompt into multiple-choice questions, such as "What color is the cube?" for a prompt about a cube. A VQA model answers them, and the score is its accuracy. TIFA v1.0 has 4,081 prompts and 25,829 questions in 12 categories.
• DSG (Cho et al. 2023) makes the questions atomic and links them in a dependency graph. If the answer to "Is there a motorcycle?" is no, the dependent question "Is the motorcycle blue?" is skipped instead of being answered with a meaningless yes. DSG-1k has 1,060 prompts.
• GenEval (Ghosh et al. 2023) uses no language model at all. Its 553 templated prompts test six skills, which are one object, two objects, counting, colors, relative position, and attribute binding. An object detector and a CLIP color classifier check each of 4 images per prompt. It agreed with human annotators 83 percent of the time, against 88 percent agreement between annotators and 80 percent for CLIP score with a threshold.
• VQAScore (Lin et al. 2024) asks one question, Does this figure show '{prompt}'? Please answer yes or no., and reports the VQA model's probability of "Yes". It set the state of the art for agreement with human judgments on eight image-text alignment benchmarks, and their own answering model, CLIP-FlanT5, beat even baselines built on GPT-4V. Their GenAI-Bench has 1,600 compositional prompts with more than 15,000 human ratings.

These benchmarks fail in their own ways. A VQA model that tends to answer yes inflates every score. Detectors trained on photographs misread generated images. Templated prompts such as "a photo of a red bench and a blue cake" look nothing like what users type, so a model can saturate a benchmark without getting better at real requests. And once a benchmark is used to choose models, it starts to shape training decisions. When a multimodal model acts as the judge instead, all the biases of language-model judges apply, such as favoring the first answer shown (§22.4). In practice, report adherence per skill, and check the automatic judge against a few hundred human labels of your own model's images.

Evaluating video

Video adds time to every question above. FVD (Unterthiner et al. 2018) is FID with a video network in place of Inception. Its features come from I3D, a video classifier made by inflating an image network's 2D filters into 3D, trained to recognize the 400 human actions of the Kinetics-400 dataset (§29.3). The paper used I3D's final logits, and later code often uses its last pooling layer. FVD is the Fréchet distance between the features of real and generated clips of a fixed length, in later work most often 16 frames.

Every flaw of FID carries over, and two more appear. Clips are expensive to generate, so FVD is often computed on a few thousand samples, where the bias is large. And actions can largely be recognized from single frames, so I3D's features care more about what is in each frame than about how it moves.

VBench (Huang et al. 2023) gave up on a single number. It scores a model on 16 dimensions, each with its own measurement and prompt suite, and checks each against human preference annotations. The dimensions fall in three groups.

• Temporal quality. Subject consistency (similarity of DINO features, from a self-supervised image encoder, across frames), background consistency (similarity of CLIP features), temporal flickering, motion smoothness, and dynamic degree. Dynamic degree is the amount of motion, measured with RAFT, a model that estimates how each pixel moves between frames.
• Frame quality. Aesthetic quality, from the LAION aesthetic predictor, a small model trained on human ratings of how good an image looks. Imaging quality, from MUSIQ, a model trained on human ratings of distortions such as noise and blur.
• Adherence. Object class, multiple objects, color, spatial relationship, human action, scene, appearance style, temporal style, and overall consistency with the prompt.

The dimensions trade off against each other. A model that barely moves scores well on consistency and flicker for free, which is why dynamic degree is measured alongside them. The VBench authors found exactly this pattern in LaVie, which excelled at consistency and flicker while producing relatively static scenes. So always read consistency scores together with dynamic degree. §34.5 describes the failures these dimensions target.

People, and models of people

Every automatic metric stands in for a judgment that people make, so the final arbiter is a human study. The most reliable design is the one used for chat models (§22.7), a blind side-by-side comparison. Show two images for the same prompt, in random left-right order, and ask which is better on one stated criterion. Ask about adherence, visual quality, and overall preference separately, because they disagree.

The rest of the design is about removing luck and bias. Use a prompt set spread over the categories you care about, such as PartiPrompts (Yu et al. 2022), with more than 1,600 prompts in 12 categories, or DrawBench (Saharia et al. 2022), with 200 prompts in 11 categories. Use several seeds per prompt. When the two models share a noise space, use the same seeds for both, so that differences come from the models and not from the luck of the draw. Record ties, and check raters with items whose answer is known.

How many votes do you need? Ignore ties for a moment. A decisive vote is then a coin flip that lands on model B with some probability $p$, and the question is whether $p$ differs from one half.

The cell checks the count by simulating 20,000 studies of 785 votes each.

The simulated power is 0.814, a little above the 0.8 we asked for, because the count is rounded up and the formula uses a normal approximation.

People are slow and expensive, so the field trains models to imitate them. Each is a Bradley-Terry reward model (§16.3), a network $r(\mathbf{c}, \mathbf{x})$ that gives each image a score for its prompt. It is trained so that $\operatorname{sigmoid}\big(r(\mathbf{c}, \mathbf{x}_A) - r(\mathbf{c}, \mathbf{x}_B)\big)$ predicts the probability that people prefer image $A$ to image $B$ for prompt $\mathbf{c}$. Three are widely used.

• PickScore (Kirstain et al. 2023) fine-tunes the CLIP-H model on Pick-a-Pic, more than 500,000 preferences from real users of a generation web app. It predicts held-out preferences with 70.5 percent accuracy, against 68.0 percent for expert human annotators. In a human study of models prompted with COCO captions, its ranking of the models correlated 0.917 with human preference, while FID's correlated $-0.900$.
• ImageReward (Xu et al. 2023) learns from 137,000 expert comparisons.
• HPSv2 (Wu et al. 2023) learns from 798,090 human choices among 433,760 images.

Public arenas pool votes across many models and fit ratings to them, exactly as §22.7 describes. A Bradley-Terry rating gives each model a strength, with the chance that A beats B equal to the sigmoid of their difference, and an Elo rating is an online version of the same idea. GenAI-Arena (Jiang et al. 2024) runs one for image generation, image editing, and video. Voters choose among four answers, A better, B better, a tie, or both bad. The best multimodal judge it tested, GPT-4o, matched the human vote only 49 percent of the time, averaged over the three tasks.

Reward models are also training targets. §33.5 showed how Diffusion-DPO and reward fine-tuning push a generator toward higher preference scores. Once a metric is optimized, it stops measuring what it used to measure. This is the reward hacking of §16.5. A model tuned on PickScore will look better on PickScore than people find it. So evaluate it with a different reward model, a held-out human study, or both.

Reading and reporting numbers honestly

Here is the whole toolbox in one table.

A report someone else can trust states the protocol in full. That means the feature network, the library and its version (for example clean-FID or torchmetrics), the reference statistics and how many images they use, $N$ for the generated set, the resizing and file format, and the guidance scale, sampler, steps, and seeds. It also gives an interval. For FID the honest interval comes from repeating the whole evaluation with independent seeds and reporting the spread. A single 50,000-sample FID has small variance, but its bias differs between models, so the spread alone does not settle a close comparison. Pair the distribution metric with precision and recall or density and coverage, and with an adherence benchmark broken down by skill. For the final decision, run a blind human study sized by the calculation above.

Computing the statistics in one pass

In production code the statistics are accumulated over batches of features as the images are generated, so no array of size $N\times d$ is ever stored. The one-pass formula $\hat{\boldsymbol\Sigma} = \big(\sum\boldsymbol\phi\boldsymbol\phi^\top - N\hat{\boldsymbol\mu}\hat{\boldsymbol\mu}^\top\big)/(N-1)$ subtracts two large numbers. When the features' mean is large compared with their spread, most of the digits cancel, and sums kept in float32 lose the answer. The block runs the same features through float32 and float64 sums at three ratios of mean to spread.

At a mean ten times the spread the two precisions agree. At 100 times, float32 is off in the second decimal, and at 1,000 times it is off by a factor of six. Summing in float64 costs almost nothing, so do it.

With real images, use a maintained implementation rather than your own feature extractor, because the Inception weights and the resizing are part of the metric.

The same habits carry beyond images. Speech generation is judged by mean opinion scores from listening tests and by automatic proxies with the same weaknesses (§37.6). Evaluating control and editing, whether an edit follows the instruction while the rest of the image stays put, returns in §34.7.

Read more

• Theis, van den Oord & Bethge (2016). A note on the evaluation of generative models. Why likelihood, samples, and nearest-neighbor tests measure different things, with the mixture argument in this section.
• Heusel, Ramsauer, Unterthiner, Nessler & Hochreiter (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. The paper that introduced FID, with its experiments on increasing image disturbances.
• Bińkowski, Sutherland, Arbel & Gretton (2018). Demystifying MMD GANs. Introduces KID, shows FID's bias and its reversals of model rankings, and proves that FID has no estimator that is unbiased for every distribution.
• Kynkäänniemi, Karras, Laine, Lehtinen & Aila (2019). Improved Precision and Recall Metric for Assessing Generative Models. The $k$-nearest-neighbor precision and recall, read with Naeem et al. (2020), Reliable Fidelity and Diversity Metrics for Generative Models, for density, coverage, and their exact expected values.
• Parmar, Zhang & Zhu (2022). On Aliased Resizing and Surprising Subtleties in GAN Evaluation. How resizing filters and JPEG compression move FID, and the clean-FID library.
• Kynkäänniemi, Karras, Aittala, Aila & Lehtinen (2023). The Role of ImageNet Classes in Fréchet Inception Distance. What FID actually looks at, and how to lower it without better images.
• Stein et al. (2023). Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. The largest human study of generated images, the case for DINOv2 features, and why memorization metrics fail.
• Jayasumana et al. (2024). Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CMMD, with the Muse experiment in which FID and human raters disagree.
• Ghosh, Hajishirzi & Schmidt (2023). GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. The detector-based adherence benchmark, read with Lin et al. (2024), Evaluating Text-to-Visual Generation with Image-to-Text Generation, for VQAScore and GenAI-Bench.

Watch next

▶

▶

▶