An image tokenizer is the network that turns an image into the short list of numbers a generator works with. It is an autoencoder. An encoder compresses a 256 by 256 photo into a small grid, and a decoder turns the grid back into a photo. When each cell of the grid holds an integer, the integers are tokens. Each one is an index into a learned table of vectors called a codebook, and a Transformer can model tokens exactly as it models words. When each cell holds a few real numbers instead, the grid is a continuous latent, the space that latent diffusion models work in. This section is about how both kinds are trained and measured, and why their design limits every generator built on them.

Start with a puzzle. Two tokenizers both turn a 256 by 256 image into a grid of codebook indices. The discrete VAE (dVAE) of DALL-E (Ramesh et al. 2021) uses a 32 by 32 grid and a codebook of 8,192 entries. VQGAN (Esser et al. 2021) uses a 16 by 16 grid, four times fewer tokens, and a codebook of 16,384. To compare them, reconstruct every image in the ImageNet validation set and compute the reconstruction FID. It runs the originals and the reconstructions through a fixed pretrained image network, fits a Gaussian to each set of features, and measures the distance between the two Gaussians. Lower is better, and 0 means the network's feature statistics cannot tell the two sets apart. The dVAE scores 32.01 and VQGAN 4.98 (VQGAN paper, Table 5). The tokenizer with a quarter of the tokens makes reconstructions that look far more real.

The same table holds a control experiment. A VQGAN trained with DALL-E's own quantizer, on the same 32 by 32 grid of 8,192 codes, scores 1.49. So the quantizer is not what separates them. The control does not fix everything, since the dVAE was trained on DALL-E's web images and has a different architecture. But the difference the VQGAN authors credit, and the subject of this section, is the training loss. The dVAE, like the VQ-VAE of §32.5 (an autoencoder that snaps each encoder vector to its nearest codebook entry), is trained with a pixel likelihood and its quantizer's own terms. Nothing in its loss judges how real the output looks. VQGAN adds two terms. A perceptual loss compares two images by the features a pretrained network extracts from them. An adversarial loss comes from a second network, a critic, trained to tell reconstructions from real photos.

The section has four parts. First, why those two losses are needed, with the game theory of the critic derived from scratch, since this is the first place the book needs it. Second, the quantizer itself, where three designs with no codebook vectors to learn, and one that quantizes in stages, have replaced the learned codebook in many systems. Third, the arithmetic of bits and tokens per image, and what each token costs the generator downstream. Fourth, the continuous latents of latent diffusion, and the open question of 2025 and 2026, why a tokenizer that reconstructs better can make generation worse.

A tokenizer is trained once and then frozen, and every image the generator ever makes passes through its decoder. Whatever that decoder cannot represent, such as small text, distant faces, or fine texture, no generator built on it can produce. That is why labs put real effort into this first stage, and why interviewers for image and video teams ask about it.

Why squared error paints textures gray

§32.1 proved that a decoder trained with squared error does best by outputting the conditional mean, the average of all the images that share its code. At high compression a code cannot pin down every detail. Take a 16 by 16 grid of codes from a codebook of $16{,}384 = 2^{14}$ entries. Each code carries 14 bits, so the image gets 3,584 bits. The image has $256\cdot256\cdot3 = 196{,}608$ pixel values, so that is about 0.018 bits per value. The code can say "grass here, sky there", but not where each blade of grass falls. Many sharp images share the code, and their average is a smooth wash.

The factor of 2 is not a coincidence of this example. It is the exact price of the natural way to make outputs look real, which is to draw the missing detail at random.

In plain words, a decoder whose outputs are indistinguishable from real images must invent the details the code does not carry. Inventing them at random from the right distribution costs exactly twice the squared error of the blurry average. The cell checks this on one pixel value per image. The code keeps only the nearest integer. The first decoder outputs the average of all values that share a code. The second outputs another real value drawn from those that share the code.

The sampling decoder's error, 0.154, is twice the mean decoder's 0.077. Its outputs also have exactly the spread of the data, 1.001, while the averages are pulled in to 0.962. Averaging always shrinks the spread, which is the blur in one number. This tension is general. Blau and Michaeli (2018) proved that lower distortion forces less realistic outputs for every distortion measure, not only squared error. Yan et al. (2021) showed that, at a given bit rate, perfect realism costs exactly a doubling of the lowest achievable squared error. So a loss that only measures reconstruction error will always choose the blur. A tokenizer that should produce sharp images needs a term that rewards realism directly. That term is the adversarial loss, and the perceptual loss makes the remaining reconstruction term more forgiving of plausible detail.

Comparing images inside a network

Squared error treats every pixel on its own, so a texture shifted by one pixel looks as wrong as noise. A perceptual distance compares two images through the features a trained network extracts from them. Those features respond to edges, textures, and object parts rather than to exact pixel positions, so two images with the same texture in slightly different places come out close.

In plain words, normalize each layer's features, weight the channels that people care about, and average the squared differences over positions and layers. Zhang et al. (2018) found that distances of this form agree with human judgments far better than PSNR (peak signal-to-noise ratio, a rescaled pixel squared error) or SSIM (a structural similarity score). The effect held across architectures and even for networks trained without labels. VQGAN's public code uses VGG-based LPIPS plus an L1 pixel term as its reconstruction loss.

A perceptual loss alone is not enough, for two reasons. It is still a regression loss, so for a code shared by many images the decoder still hedges, now in feature space, and the result is less blurry but not sharp. And the decoder can learn small patterns that move the network's features without looking right to a person. These are the high-frequency artifacts that perceptual-loss training is known to produce. The critic is what removes the hedge.

A critic that judges whether an image looks real

A generative adversarial network, or GAN (Goodfellow et al. 2014), trains two networks against each other. A discriminator, or critic, learns to tell real images from generated ones. A generator learns to fool it. In a tokenizer the generator is the whole autoencoder, so the fakes are reconstructions $\hat{\mathbf{x}} = \operatorname{Dec}_\theta(\operatorname{Q}(\operatorname{Enc}_\phi(\mathbf{x})))$ of real images, where $\operatorname{Dec}_\theta$ is the decoder with parameters $\theta$. We write $p_\theta$ for the distribution of those reconstructions. This is the first place the book needs GANs, so we derive the pieces we use. They return for audio in GAN vocoders.

In plain words, the critic gets a reward for calling real images real and fakes fake, and the generator gets the opposite reward. What is the generator minimizing? To find out, fix the generator, find the best critic, and substitute it back. That needs the Jensen-Shannon divergence of §0.5, a symmetric and bounded relative of the KL divergence, restated in the theorem below.

In plain words, a perfect critic outputs the fraction of the local probability mass that is real. Against that critic, the generator is minimizing the Jensen-Shannon divergence between real and generated images. The minimum, $-\log 4$, is reached only when $q = p$, where the best critic outputs one half everywhere. This is a statement about the ideal game. Real critics are networks trained for a few steps at a time, far from $\operatorname{Disc}^\star$, which is part of why GAN training needs care.

The cell checks the theorem on a grid for two Gaussians, one fixed and one shifted. It computes $\log(1 - \operatorname{Disc}^\star)$ as $\log q - \log(p + q)$. Forming $1 - \operatorname{Disc}^\star$ in floating point rounds to 0 wherever $q$ is tiny compared with $p$, and the logarithm would then return $-\infty$.

With no shift the value is $-\log 4 = -1.3863$. As the two Gaussians separate, the Jensen-Shannon divergence climbs toward $\log 2 = 0.6931$ and the value climbs toward 0, the score of a critic that is never wrong.

import numpy as np

def optimal_discriminator_value(p, q, dx):
    raise NotImplementedError

Two failures of the ideal game come up in every GAN interview. The first is vanishing gradients from disjoint supports. Real images and early generator outputs typically sit on thin, low-dimensional sets that do not overlap. Then $\operatorname{JSD}$ is the constant $\log 2$ however far apart the sets are, and a near-perfect critic gives the generator almost no useful gradient. The second is mode collapse, where the generator covers only a few kinds of data. In alternating training it can win a round by moving all its mass to whatever the current critic rates highest. A tokenizer is largely protected from mode collapse, because its generator reconstructs a given input and the reconstruction losses tie each output to that input.

Why the original generator loss stalls, and the two losses used instead

In the minimax game, the generator minimizes $\mathbb{E}_{q}[\log(1 - \operatorname{Disc}(\hat{\mathbf{x}}))]$. Early in training the critic spots fakes easily, so $\operatorname{Disc}(\hat{\mathbf{x}}) = \sigma(a)$ is near 0 and the logit $a$ is very negative. The derivative of the generator's loss with respect to that logit is $\frac{d}{da}\log(1 - \sigma(a)) = -\sigma(a)$. It is nearly 0 exactly when the generator is doing worst. This is called saturation. Goodfellow et al. (2014) suggested that the generator instead maximize $\log\operatorname{Disc}(\hat{\mathbf{x}})$, that is, minimize $-\log\sigma(a)$. Its derivative, $-(1 - \sigma(a))$, is close to $-1$ in the same situation. Both losses share the same fixed point. Only the gradient far from it differs.

At $a = -8$ the minimax gradient is $-0.00034$ and the non-saturating one is $-0.99966$, about 3,000 times larger. Modern tokenizers mostly use a third choice, the hinge loss.

In plain words, the hinge critic stops pushing on an example once it is classified correctly with a margin of 1, like a support vector machine. The generator just raises the critic's score of its outputs. Miyato et al. (2018) paired the hinge loss with spectral normalization, which divides each weight matrix of the critic by its largest singular value (§0.10) so that no layer can stretch its input by more than a factor of 1. VQGAN's public code trains with the hinge loss, even though its paper writes the log-loss form.

The widget below puts the theorem and the three losses side by side. It draws the optimal critic for a data density and a generator density on a line. One slider moves the generator and a second picks one fake sample from it. Bars show the pull of each loss on that sample, pointing the way the sample would move. All three use the same optimal critic, so only the loss differs.

With Gaussian tails the two densities always overlap. So the optimal logit $a^\star = \log p - \log q$ keeps a slope everywhere. The non-saturating and hinge losses keep a derivative near $-1$ in that logit and keep pulling, and only the minimax pull dies. With bounded supports pulled apart, the divergence sticks at $\log 2$ and the optimal critic is exactly 0 on the whole fake support. Its logit is flat there, and every loss gives zero pull. A changed loss cures saturation but not disjoint supports. That second failure is why the WGAN and the critic regularizers below limit the critic's slope. A critic with bounded slope cannot jump from real to fake, so its score keeps rising along the path the fake mass must travel.

A critic for texture that judges patches

A discriminator that outputs one number per image has to judge global layout and fine texture at once. The reconstruction losses already take care of layout. What they get wrong is local texture. So tokenizers use a patch discriminator, the PatchGAN of Isola et al. (2017). It is a fully convolutional network that outputs a grid of logits. Each logit sees only one window of the image and asks whether that patch looks real, which is exactly the question the stripe example needs answered. The loss is averaged over the grid. The window is the receptive field of an output unit, the set of input pixels it depends on. The arithmetic from §0.8 gives the 70 by 70 pixel windows of the original design.

So a 256 by 256 reconstruction is judged by 900 overlapping critics. Each looks at a 70 by 70 window, and neighboring windows are 8 pixels apart.

GAN terms are the unstable part of tokenizer training, and three habits keep them in check.

1. Warm up without them. The autoencoder first learns to reconstruct, and the adversarial term is switched on later. The tokenizer of LlamaGen, an autoregressive image generator, enables it after 20,000 iterations, with weight 0.5 (Sun et al. 2024).
2. Regularize the critic. Mescheder et al. (2018) showed that unregularized GAN training need not converge when the distributions lie on low-dimensional sets, as images do. A penalty on the critic's gradient at real images, $R_1 = \tfrac{\gamma}{2}\mathbb{E}_{p}\lVert\nabla_{\mathbf{x}} a_\psi(\mathbf{x})\rVert^2$ with a weight $\gamma$, restores convergence near the equilibrium. The image and video tokenizer MAGVIT-v2 (Yu et al. 2023), whose quantizer appears later in this section, uses this $R_1$ penalty, with weight 10. It adds the LeCAM penalty (Tseng et al. 2021) on top, with weight 0.001. LeCAM pulls the critic's outputs on real images toward a moving average of its outputs on fakes, and the reverse, so the two cannot drift far apart.
3. Balance the two gradients. Scale the adversarial gradient to match the reconstruction gradient, which is VQGAN's adaptive weight.

The VQGAN objective and its adaptive weight

VQGAN is a VQ-VAE trained with all of the pieces above. Recall from §32.5 that a VQ-VAE needs two extra terms beside the reconstruction loss. The codebook loss pulls each chosen code toward the encoder output it replaced. The commitment loss keeps the encoder output close to its code. Both use the stop-gradient $\operatorname{sg}[\cdot]$, which is the identity going forward and has zero derivative going back (.detach() in PyTorch), so each term moves only one side.

Here is what the ratio does, first with numbers. Suppose at some step the reconstruction gradient at the last layer has norm 0.02 and the adversarial gradient has norm 0.5. Then $\lambda_{\text{adv}} = 0.04$, and the weighted adversarial gradient has norm $0.04 \times 0.5 = 0.02$, the same as the reconstruction gradient. In general, write $\mathbf{g}_{\text{rec}} = \nabla_{G_L}\mathcal{L}_{\text{rec}}$ and $\mathbf{g}_{\text{GAN}} = \nabla_{G_L}\mathcal{L}_{\text{GAN}}$. With $\delta = 0$, $\lVert\lambda_{\text{adv}}\mathbf{g}_{\text{GAN}}\rVert = \frac{\lVert\mathbf{g}_{\text{rec}}\rVert}{\lVert\mathbf{g}_{\text{GAN}}\rVert}\lVert\mathbf{g}_{\text{GAN}}\rVert = \lVert\mathbf{g}_{\text{rec}}\rVert$.

In plain words, at the last layer the critic is always allowed to push exactly as hard as the reconstruction loss, however the scale of either loss drifts during training. Without it, the adversarial gradient can be orders of magnitude larger or smaller than the reconstruction gradient, depending on how confident the critic is. A fixed weight would be right for only one phase of training.

Three practical points matter in an implementation or an interview.

• Why the last layer. It is the layer nearest the output, so both gradients are cheap to get. Each needs one backward pass through its loss network (VGG for LPIPS, or the discriminator) to the decoder's output and through that one layer, never through the rest of the decoder or the encoder. The choice is a heuristic, not an optimality result. It balances gradient sizes at that layer only, and says nothing about directions or deeper layers.
• Clamping. The public code clamps the ratio to $[0, 10^4]$ so that a vanishing $\mathbf{g}_{\text{GAN}}$ cannot blow it up, adds $10^{-4}$ rather than $10^{-6}$ in the denominator, and multiplies the result by a fixed factor.
• Detaching. Because $\lambda_{\text{adv}}$ must be a constant, it is computed with torch.autograd.grad on the two losses separately and detached, before the combined loss is backpropagated. The PyTorch block at the end of the section shows this.

import numpy as np

def hinge_d_loss(real_logits, fake_logits):
    raise NotImplementedError

def hinge_g_loss(fake_logits):
    raise NotImplementedError

Quantizers without a codebook to learn

Recall the main weakness of a learned codebook from §32.5. A code that drifts away from the encoder outputs stops being anyone's nearest code. It then stops receiving gradient and never moves again, a dead code. Large codebooks end up mostly unused unless you add restarts, low-dimensional normalized lookups, or other fixes. In 2023 two groups asked a blunter question. What if there were no code vectors to learn at all, only a fixed grid that the encoder learns to fill?

Finite scalar quantization

The simplest fixed grid rounds each number separately. Squash each channel of the encoder output into a fixed range, round it to one of a few levels, and read the levels as the digits of the token id. With 3 channels of 5 levels each there are $5^3 = 125$ grid points, so 125 possible tokens, and nothing to learn but the encoder.

In plain words, each channel is a dial with $L_j$ positions, and the token id is the reading of all the dials as one number, the way a units digit 3, a tens digit 2, and a hundreds digit 1 read as 123, with channel 1 as the units digit. There is no codebook vector to learn, no codebook loss, no commitment loss, and nothing that can die. The offset for even $L_j$ is the detail people get wrong. Take $L = 8$. Bounding with $\lfloor L/2\rfloor\tanh(z)$ gives the range $(-4, 4)$, which rounds to nine values, $-4$ to $4$. The definition instead uses the half-width $(L - 1)/2 = 3.5$ and shifts down by half a level. The range becomes $(-4, 3)$, which rounds to exactly eight values.

Mentzer et al. (2023) recommend at least 5 levels per channel. They give level sets such as $[8, 5, 5, 5]$ for about $2^{10}$ codes and $[8, 8, 8, 6, 5]$ for about $2^{14}$ (15,360 codes, 13.91 bits). With codebooks that size, FSQ used close to 100 percent of its codes in their image-generation (MaskGIT) and dense-prediction (UViM) experiments, without any auxiliary loss, while VQ's usage fell as the codebook grew. The reason is that no code has a vector that could drift away from the data, so no code can be starved. And the straight-through gradient reaches the encoder whatever level is chosen. An interviewer will ask exactly this, why FSQ codes cannot die, and that is the answer. Codes can still go unused if the encoder's outputs bunch up, but none is stuck for good.

import numpy as np

def fsq_quantize(z, levels):
    raise NotImplementedError

def fsq_index_to_codes(idx, levels):
    raise NotImplementedError

Lookup-free and binary spherical quantization

Push FSQ to its extreme, two levels per channel. Each channel then keeps only its sign, and the token id is the string of signs read as bits. This is lookup-free quantization (LFQ), the quantizer of MAGVIT-v2 (Yu et al. 2023). A codebook of $K = 2^d$ codes needs only $d = \log_2 K$ channels. For example, $\mathbf{z} = (0.3, -1.2, 0.7)$ has signs $(+, -, +)$, bits $(1, 0, 1)$, and index $1 + 0\cdot2 + 1\cdot4 = 5$ out of 8.

In plain words, the first term asks each latent to commit to its code, so its own entropy is low. The second asks the batch as a whole to spread over many codes, so the entropy of the average is high. The exact penalty is lowest, $-d\log 2$, when every latent is certain of its code and the batch uses all $2^d$ codes equally, which needs at least $2^d$ latents. The factorized form in the box reaches the same minimum as soon as every channel is $+1$ for half of the batch, which two complementary codes already achieve. That blind spot is the price of the approximation explained in the box below. In exchange, the factorized form costs $Bd$ numbers rather than the $B\cdot2^d$ the exact batch term needs, 262,144 per latent at $d = 18$.

MAGVIT-v2 used $K = 2^{18}$, about 262,000 codes, and found that, unlike VQ, both reconstruction and generation kept improving as the vocabulary grew. To help a 307-million-parameter masked Transformer predict from so many classes, it split each token into two sub-tokens of $2^9$ values each. That split is possible precisely because an LFQ index is a string of bits.

LFQ's error grows without bound as $\lVert\mathbf{z}\rVert$ grows, because every output has the same length $\sqrt d$. Binary spherical quantization adds one step that fixes this.

Zhao et al. report that BSQ trains faster than LFQ and reaches a better tokenizer. One convention to watch when you compare quantized outputs across code bases is that BSQ maps $\operatorname{sign}(0)$ to $+1$ and LFQ to $-1$. BSQ's index still counts only strictly positive entries, as LFQ's does.

import numpy as np

def lfq(z):
    raise NotImplementedError

def lfq_entropy_penalty(z, tau=1.0):
    raise NotImplementedError

The same bits, three partitions

A quantizer at $b$ bits cuts the latent space into at most $2^b$ cells, and the decoder outputs one point per cell. For a fixed partition, the squared-error-optimal point is the mean of the latents in the cell. So the quantizers differ only in which partitions they can draw. The next cell quantizes the same 4,000 two-dimensional Gaussian latents at 6 bits, 64 cells, three ways.

• VQ with a k-means codebook of 64 codes.
• FSQ with levels $[8, 8]$, at an input scale of 1.4 found by a quick search, which a real encoder would learn.
• LFQ, which needs 6 channels, so the 2-D latent is first lifted by a random affine map (a linear map plus an offset).

The floor is Shannon's distortion-rate bound. For a 2-D standard Gaussian coded at $R$ bits per latent, the least possible squared error summed over both coordinates is $2\cdot2^{-R}$ (each coordinate gets $R/2$ bits and error $2^{-R}$). That is the reverse water-filling of §32.4 with equal variances, and no quantizer of any kind can beat it. The cell also reports perplexity, the exponential of the entropy of code usage, which is the effective number of codes in use.

The learned codebook wins on distortion, 0.056 against FSQ's 0.074. K-means can place and shape its cells freely to fit the density, while FSQ's cells are axis-aligned boxes of a fixed grid. The tanh already widens FSQ's outer cells, and the only free parameter here is the input scale. Both use all 64 codes. LFQ through an affine map loses by a factor of eight and reaches only 20 of its 64 codes. Six lines in a plane cut it into at most $1 + 6 + \binom{6}{2} = 22$ regions (with no offset, all six lines pass through the origin and give at most 12). Sign quantizers are efficient only when every channel is an independent degree of freedom. That is why real LFQ tokenizers give the quantizer 18 channels from a deep nonlinear encoder rather than a few. None of the three reaches Shannon's 0.031, which assumes coding long blocks of latents jointly, something a per-token quantizer never does.

This toy freezes the latents, which is the one thing a real tokenizer does not do. In training, the encoder reshapes its outputs to suit the quantizer, spreading them over FSQ's grid or LFQ's corners. That is why fixed partitions work as well as they do at scale. There, the learned codebook's flexibility, which wins here, matters less than its training problems. The widget lets you explore the partitions yourself, including residual VQ, the subject of the next part. It uses 1,200 latents and spreads LFQ's lines at even angles instead of drawing them at random, so its counts differ a little from the cell's. At 6 bits its lines reach 21 of the 22 possible regions.

Residual quantization

Residual quantization gets many codes from small codebooks by quantizing in stages. Quantize the latent. Then quantize what the first code missed. Then quantize what is still missing, and so on. The reconstruction is the sum of the chosen code vectors.

Each stage is a nearest-neighbor search over $K$ codes, so encoding costs $n_q K$ distance computations. Yet the number of distinct reconstructions can be as large as $K^{n_q}$. That is a codebook of $K^{n_q}$ entries that you never store or search. Product quantization in §7.4 gets the same exponential count a different way, by splitting the dimensions instead of stacking residuals.

So it is tempting, and wrong, to say that every stage reduces the error. It is true on average for a codebook fit to the residuals, and false for individual vectors. The difference matters in interviews and in code reviews.

In plain words, a stage fit to its residuals helps on average, since at worst it could have output their mean. But a particular vector gets worse when its residual is already smaller than the gap to every code, as $\mathbf{z}'$ did above. The guarantee in (b) is for the vectors the codebook was fit on. In end-to-end training with moving codebooks it holds only approximately. That is why RQ-VAE sums the commitment loss over every depth, $\sum_{j}\lVert\mathbf{z} - \operatorname{sg}[\hat{\mathbf{z}}^{(j)}]\rVert^2$ with $\hat{\mathbf{z}}^{(j)}$ the partial sum after $j$ stages. The encoder is pulled toward every partial reconstruction, so the error tends to fall stage by stage.

The cell fits four stages of 16 codes (4 bits each) to the Gaussian latents from before. It checks the average error after each stage, counts the vectors each stage made worse, and compares two stages with a single codebook of 256 codes, the same 8 bits.

The average error falls by a factor of five to ten per stage, and part (b) guarantees it can never rise. Yet at every stage about 5 percent of the vectors get worse, as part (a) warns. At 8 bits, two stages of 16 codes reach 0.027 while one codebook of 256 reaches 0.015, so stacking costs distortion. RVQ's partition is constrained, because every stage-1 cell is subdivided the same way. The price buys a search over 32 codes instead of 256. At $n_q$ stages of $K = 16{,}384$ codes, the single codebook would need $2^{14 n_q}$ entries and could not exist at all.

The bill arrives downstream. An autoregressive generator, one that predicts tokens one after another, now has $n_q$ codes to predict at every position. RQ-VAE pairs its tokenizer with an RQ-Transformer. A large spatial Transformer runs over the 64 positions, and a small depth Transformer predicts the 4 codes at each one, so the long sequence of $64 \times 4$ tokens is never formed. The best-known image use of residual quantization since then is VAR (Tian et al. 2024), which takes the residuals across scales instead of at one resolution. Its tokenizer writes an image as a pyramid of token maps, from 1 by 1 up to the full 16 by 16 grid, all drawn from one shared codebook of 4,096. Each map quantizes what the upsampled coarser maps still miss, so a generator can predict one whole map at a time, coarse to fine. Autoregressive image generation defines it and its next-scale prediction. Residual quantization is also the workhorse of neural audio codecs, which train with a random number of stages so that one model serves many bitrates.

import numpy as np

def rvq_encode(Z, codebooks):
    raise NotImplementedError

def rvq_decode(idx, codebooks):
    raise NotImplementedError

Bits, tokens, and what the generator pays

Every tokenizer is a point in a three-way trade-off, which Dieleman (2025) calls rate, distortion, and modelability. Rate is how much information the tokenizer keeps. Distortion is how far its reconstructions are from the input. Modelability is how easy its tokens or latents are for a generator to learn. The first two are the rate-distortion pair of §32.4. The third is new, and it decides what a tokenizer is worth. This part counts the first and the cost of the third.

For continuous latents, the stored bits overstate the information. What the latent tells the decoder about the image is the mutual information $I_q(\mathbf{x};\mathbf{z})$. By the decomposition of the KL term in §32.4, it is at most the rate, the expected KL from the encoder's distribution to the prior. It is set by the encoder's noise, not by the floating-point format. We count storage only to compare sizes.

What does a token cost the generator? An autoregressive Transformer over the codes runs $T n_q$ sequential prediction steps, or fewer with a depth Transformer or a masked decoder that predicts many tokens in parallel. Each full-attention layer costs time proportional to $T^2$ for the score matrix plus $T$ for everything else (§4.7). A diffusion Transformer cuts the continuous latent into $P$ by $P$ patches (§27.1), so it sees $T/P^2$ tokens at every one of its sampling steps. Halving $f$ quadruples $T$ and multiplies the attention cost by 16. So downsampling is the most expensive design choice a tokenizer has, and the field keeps pushing $f$ up, to 32, 64, and as far as 128 in DC-AE (Chen et al. 2024). Its high-compression autoencoders gave a large diffusion Transformer (UViT-H) a 19.1 times inference speedup and a 17.9 times training speedup on ImageNet at 512 by 512, at a better FID than the usual $f = 8$ autoencoder (SD-VAE-f8).

The cell computes the budget of published tokenizers, beside their published reconstruction FID.

Read the table with care, because the FIDs come from different papers. The latent diffusion (LDM) autoencoders were trained on OpenImages, and DALL-E's dVAE on DALL-E's 250 million web image-text pairs. The VQGAN, LlamaGen, RQ-VAE, and TiTok tokenizers were trained on ImageNet. All rows except SD3 are measured on ImageNet validation, and the SD3 paper names neither the training data nor the evaluation set of its autoencoders. Two rows still make a sharp point. VQGAN and LlamaGen at $f = 16$ spend exactly the same 3,584 bits, yet LlamaGen's reconstruction FID is 2.19 against 4.98. The main reason is that LlamaGen looks up codes in an 8-dimensional, $\ell_2$-normalized space, a fix from §32.5. Its Table 2 shows usage rising from 0.29 to 97 percent as the code dimension falls from 256 to 8. A budget sets a ceiling, and design decides how close to it you get.

TiTok (Yu et al. 2024) goes further and drops the grid altogether. A ViT encoder (§27.2) summarizes the image into 32 learned tokens, which at a reconstruction FID of 2.21 match MaskGIT's retrained 256-token VQGAN (2.28) and beat the original 256-token VQGAN's 4.98. FlexTok (Bachmann et al. 2025) makes the length variable, from 1 to 256 tokens per image, ordered coarse to fine.

The widget puts the tokenizers of the table, plus DC-AE, on one log scale of tokens per image. Click a row to see the image cut into the tokens its generator works on. Then raise the image size and watch the token count, and the attention cost with its square, grow.

import numpy as np

def tokenizer_budget(H, W, f, K=None, c=None, n_q=1, bits_per_float=16):
    raise NotImplementedError

Continuous latents for diffusion

Diffusion models do not need discrete tokens. They need a compact space with a known scale. Latent diffusion (Rombach et al. 2022) gets one from an autoencoder trained exactly like VQGAN, with L1, LPIPS, and a patch discriminator, but with a continuous Gaussian encoder in place of the quantizer.

The KL weight is so small that the rate is essentially unconstrained. In the language of §32.4, minimizing $\mathcal{D} + \beta\mathcal{R}$ picks the point where the rate-distortion curve has slope $-\beta$. So a tiny $\beta$ sits at the high-rate, low-distortion end, where the curve is nearly flat. The KL is not there to make samples from $\mathcal{N}(\mathbf{0}, \mathbf{I})$ decode to images, since nobody samples this prior. Its job is to keep the latent's scale from drifting, so that a diffusion model, whose noise schedule assumes data of a known variance, can run on it. The scale factor finishes that job.

Here is the classic bug, with numbers. Stable Diffusion 1.x multiplies its latents by 0.18215, which divides them by their measured standard deviation of about 5.49. Forget the factor and the latents are 5.49 times larger than the noise schedule assumes. Their variance is then $5.49^2 \approx 30$ times larger, so the signal-to-noise ratio at every step is off by a factor of 30. The KL-f8, 4-channel model reconstructs ImageNet validation images at PSNR 24.19 dB and reconstruction FID 0.90 (LDM, Table 8). The VQ-regularized variants in the same paper needed no rescaling, since their latents already had variance close to 1. How a diffusion model is trained in this space is the subject of latent diffusion, and its video counterpart, a 3D causal VAE, appears in video generation.

Better reconstruction, worse generation

The obvious way to improve reconstruction is more channels, and it works. In Stable Diffusion 3 (Esser et al. 2024, Table 3), $f = 8$ autoencoders with 4, 8, and 16 channels reconstruct at PSNR 25.12, 26.40, and 28.62 dB, with reconstruction FID 2.41, 1.56, and 1.06. But a flow model trained on the 16-channel latent needed more capacity to match the others. Only at depth 22 did its gap to the 8-channel latent become negligible (their Figure 10). SD3 chose 16 channels because it planned to scale the generator. The pattern is general, and it has a name, the reconstruction versus generation dilemma.

Why should a better latent be harder to model? A higher-dimensional latent carries more of the image's fine detail. That detail is the part of the distribution closest to noise, with little structure a network can exploit. And a latent trained only to reconstruct has no reason to place similar images near each other. The generator must then learn a distribution that is both larger and less organized. This explanation is a working hypothesis, consistent with the experiments below, not a theorem. What labs did about it in 2025 is a set of empirical answers, and the field has not settled.

Scaling the tokenizer itself is not a simple win either. ViTok (Hansen-Estruch et al. 2025) found that scaling the encoder helped little for either reconstruction or generation. Scaling the decoder improved reconstruction, with mixed effects on generation, plausibly because a large decoder can invent detail the latent never stored. The same logic sits behind the argument that a representation should predict in latent space rather than reconstruct pixels at all, which the comparison of predictive and generative objectives takes up.

Discrete or continuous? Discrete tokens let one Transformer read and write images with the same cross-entropy loss it uses for text. That is exactly what the early-fusion models of §31.6 do, models that put text and image tokens in one sequence. The autoregressive and masked generators of autoregressive image generation build on them too. Continuous latents keep more information per number and suit diffusion. MAR (Li et al. 2024) showed that an autoregressive model can also predict continuous tokens, by replacing the softmax with a small per-token diffusion head. Both camps now borrow from each other, and FSQ and LFQ, which round a continuous latent, sit between them.

Choosing a quantizer

The table collects every quantizer in this chapter. The first four rows, the learned codebooks, are taught in §32.5, and the rest in this section.

In PyTorch

The block below puts the section's pieces into one training step on random tensors. It has an FSQ module with round_ste (rounding with the straight-through estimator), an LFQ module with its entropy penalty, the hinge losses, and the adaptive weight computed with torch.autograd.grad at the decoder's last layer. An L1 loss stands in for L1 plus LPIPS, which needs pretrained weights.

The weighted adversarial gradient at the last layer has the same norm as the reconstruction gradient, as the adaptive weight promises. In a real tokenizer the discriminator gets its own optimizer and steps on d_loss after the autoencoder steps. The adversarial terms are multiplied by zero until the warm-up ends, and the latent z comes from a convolutional or ViT encoder.

Read more

• Esser, Rombach & Ommer (2021). Taming Transformers for High-Resolution Image Synthesis. VQGAN, with the perceptual and patch-adversarial losses, the adaptive weight, and the reconstruction FID comparison of Table 5.
• Goodfellow et al. (2014). Generative Adversarial Networks. The minimax game, the optimal discriminator, and the Jensen-Shannon result, in a few pages.
• Zhang et al. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. LPIPS and the human-judgment data behind it.
• Mentzer et al. (2023). Finite Scalar Quantization: VQ-VAE Made Simple. FSQ, its level recommendations, and the usage comparison with VQ.
• Yu et al. (2023). Language Model Beats Diffusion: Tokenizer is Key to Visual Generation. MAGVIT-v2, lookup-free quantization, and the entropy penalty.
• Lee et al. (2022). Autoregressive Image Generation using Residual Quantization. RQ-VAE and RQ-Transformer, with the depth versus codebook-size ablation.
• Yao, Yang & Wang (2025). Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. The clearest measurement of the dilemma, and the foundation-model alignment fix.
• Dieleman (2025). Generative modelling in latent space. An essay on rate, distortion, and modelability that frames this whole section.

Watch next

▶

▶

▶

▶