This section is about generating an image the way a language model writes text, one piece at a time, with each piece predicted from the pieces already written. It also covers two relatives that write many pieces per step, one by filling in a partly hidden picture and one by sharpening a picture from coarse to fine. These generators matter for two reasons. They are the main alternative to the diffusion models of §34.1 to §34.3, and they are how a language model itself learns to draw. OpenAI describes the image generator inside GPT-4o as an autoregressive model (OpenAI 2025).

The pieces are usually image tokens. An image tokenizer (§32.5 and §32.6) is an autoencoder whose encoder turns an image into a small grid of integers and whose decoder turns such a grid back into pixels. Each integer is the index of one entry in a learned table of vectors, the codebook. A 256 by 256 image becomes 256 or 1,024 integers, a sentence in a language of about 1,000 to 16,000 words (§2.7 met image tokens as a vocabulary). Language models are very good at long sequences of integers. So train a GPT, a Transformer that predicts the next token, on those integers, and you have an image generator. DALL-E (Ramesh et al. 2021) and Parti (Yu et al. 2022) did exactly that, and it works.

The recipe has two problems. A GPT writes a 1,024-token image one token per network call, so one picture costs 1,024 calls that must run one after another. And it writes in raster order, row by row and left to right, an order that has nothing to do with how images are structured.

We follow that road and the two ways off it. We start with the chain rule of probability applied to pixels, then to tokens, with real systems counted. We sample image tokens, including classifier-free guidance applied to logits. Then comes masked decoding, which writes many tokens per call in any order and pays a price we can compute exactly. Then comes next-scale prediction, which writes the whole image again at a finer resolution in each call and saves less work than it seems to. Then come models whose tokens are continuous vectors, and the native multimodal models that put all of this inside a language model. We end with a side-by-side comparison with diffusion.

The chain rule, applied to pictures

The chain rule of probability says that the probability of several things together is the probability of the first, times the probability of the second given the first, times the probability of the third given the first two, and so on. It holds for any order you pick. §3.1 used it to define a language model, where the things are words. For an image the things are pixel values. Flatten the image into a sequence in raster order, row by row and left to right within a row. For a color image, the red, green, and blue values of each pixel come one after another. Then apply the same rule. Here it is on the smallest image that has any structure.

In plain words, predict each pixel value from the ones before it, multiply the predictions to get the probability of the whole image, and generate by asking for one value at a time.

Two properties make this family attractive. The first is that the likelihood is exact. A VAE (§32.3) and a diffusion model trained with its usual loss (§33.2) give only a lower bound on it. With an exact likelihood, models can be compared by how much probability they give to held-out images. The usual unit is bits per dimension, $-\log_2 p_\theta(\mathbf{x})/d_x$. It is the number of bits per pixel value that a compressor built from the model would need (§32.3 defined it, and §0.5 explains code lengths). Lower is better, and a model that knows nothing about 8-bit images scores 8.

The second is that training is fully parallel. Call a network causal if its output at position $i$ depends only on positions before $i$. With a causal network, all $d_x$ predictions for a training image come out of one forward pass, exactly as with the next-token loss of a GPT (§6.2). Only sampling is sequential.

The next cell does the same at a size where counting still works. The data are the 100 axis-aligned filled rectangles on a $4\times4$ binary grid, each with probability $\tfrac{1}{100}$. Every conditional $p(x_i\mid x_{<i})$ is computed exactly, by counting the rectangles that agree with the prefix, and the chain rule reproduces $\log_2\tfrac{1}{100}$.

A real model does not know the distribution. It must estimate each conditional from samples. The crudest estimator is a table with one entry per prefix, holding counts of what came next. The next cell trains such a table on 2,000 samples. It smooths the counts by adding $\tfrac12$ to each, so that a prefix never seen gets probability $\tfrac12$ rather than a division by zero. Then it scores fresh images and draws 500 samples.

The first lesson is about generalization. The table stores 576 of the 65,535 possible prefixes and scores 0.427 bits per pixel, close to the true 0.415. A table works here only because the data are tiny. For real images almost every prefix is new. That is why every practical model is a network, which shares parameters across positions and can give sensible predictions for prefixes it never saw.

The second lesson is about sampling. Only about 88 percent of the samples are rectangles. Smoothing gives every certain step a small probability of going wrong, about $0.5/21$ for a prefix seen 20 times. Those small errors compound over the 16 steps. Once a sample leaves the set of real images, every later prediction comes from a context the model never saw in training, so it is little better than a guess.

This failure has a name. Training only on real prefixes is called teacher forcing. A model trained that way and then run on its own outputs suffers exposure bias, the problem §3.5 met in translation. A crude model shows how bad it gets for long sequences. If each step independently put probability $\delta$ off the real data, a sequence of $N$ steps would stay on it with probability $(1-\delta)^N$. For $N = 1{,}024$ and $\delta = 10^{-3}$, that is $0.36$. Real errors are neither independent nor uniform, but the lesson holds. Long raster sequences are fragile, and an interviewer will expect you to say so.

Masked convolutions and their blind spot

The deep autoregressive image models of 2016 were recurrent and convolutional networks, and the convolutional ones were faster to train. A convolution (§0.8) computes each output from a small window of inputs around the same position. So a plain convolution would let the prediction for pixel $i$ see pixel $i$ itself and the pixels after it, which is cheating. The fix is a masked convolution, which zeroes every kernel weight that would read the current pixel or a later one. It is the core of PixelCNN (van den Oord et al. 2016).

There are two masks. The first layer uses mask A, which also hides the center, so the output for pixel $i$ never sees $x_i$. Later layers use mask B, which keeps the center. That is safe because by then the feature at position $i$ is already a function of $x_{<i}$ alone. The cell traces which input pixels one output can depend on, after one mask-A layer and nine mask-B layers.

The output pixel x sees everything marked #. It never sees the wedge marked . to its upper right, however deep the stack. The receptive field of an output is the set of inputs it can depend on. Each layer can extend it up one row and right one column, so a pixel $k$ rows up is visible only within $k$ columns to the right. This gap is the blind spot, and the model cannot use those pixels however much they would help.

The fix splits the network into two stacks (van den Oord et al. 2016, the Gated PixelCNN). A vertical stack sees all the rows above. A horizontal stack sees the current row to the left. Together they cover every earlier pixel. The same paper also replaced each ReLU with a gate, $\tanh(\mathbf{W}_f * \mathbf{h})\odot\operatorname{sigmoid}(\mathbf{W}_g * \mathbf{h})$, where $*$ is convolution and $\odot$ is the elementwise product. The sigmoid decides how much of each tanh feature passes. Transformers have no blind spot at all, because a causal attention mask (§4.4) lets every position read every earlier one directly.

The largest pixel models were Image GPT (Chen et al. 2020), GPT-2-style Transformers of up to 6.8 billion parameters trained directly on pixels. To keep sequences short, it downsampled images to $32\times32$, $48\times48$, or $64\times64$. It also replaced each RGB triple by one of 512 colors found by k-means clustering. So a $32\times32$ image is a sequence of 1,024 tokens instead of 3,072.

That is the wall pixel models hit. A $256\times256$ RGB image is $196{,}608$ values. A pixel model needs $196{,}608$ sequential network calls per image, and attention over a sequence that long is expensive even in training. Most of those values are detail nobody perceives, the argument §34.1 made for latent diffusion. The fix is the same one latent diffusion uses. Compress the image first with an autoencoder, then model the compressed code.

Generating tokens instead of pixels

Here is the tokenizer in symbols. A vector-quantized tokenizer (§32.5) encodes an image into an $h\times w$ grid of vectors. It replaces each vector by the index of its nearest entry in a codebook $\mathbf{C}_{\text{vq}} \in \R^{K\times d_z}$ of $K$ entries, each of width $d_z$. The most widely used design adds perceptual and adversarial losses to the training of the tokenizer, which keep the decoded images sharp. It is called VQGAN (Esser et al. 2021), and §32.6 builds it. The downsampling factor is the side of the pixel patch each token stands for. With factor 16 a $256\times256$ image becomes $16\times16 = 256$ tokens, and with factor 8 it becomes 1,024. In this section $N = hw$ is the number of image tokens.

In plain words, an image tokenizer turns pictures into sentences in a language with $K$ words, and a GPT learns that language.

In the decoder-only layout a training sequence is the caption, then a begin-of-image token that marks where the image starts, then the $N$ image tokens. The inputs are the sequence without its last element. The targets are the sequence shifted left by one, as for any language model. With every target weighted equally, a 30-token caption and a 1,024-token image make 97 percent of the targets image tokens (§31.6 does this arithmetic). So systems choose the weights on purpose. Many count the loss on image targets only. DALL-E averaged the loss over each modality separately, then weighted the text by $\tfrac18$ and the image by $\tfrac78$.

import numpy as np

def ar_training_pairs(text_ids, image_ids, boi_id, pad_id, text_len):
    raise NotImplementedError

DALL-E (Ramesh et al. 2021) was the first large system of this kind. Its tokenizer is a discrete VAE trained with the Gumbel-softmax relaxation of §32.5, a smooth stand-in for picking one code that lets gradients flow. It turns a $256\times256$ image into $32\times32 = 1{,}024$ tokens from a codebook of 8,192. The generator is a 12-billion-parameter decoder-only Transformer. Its attention is sparse, meaning each layer reads only a fixed part of the earlier positions. It reads up to 256 caption tokens, from a byte-pair vocabulary of 16,384, followed by the 1,024 image tokens. It was trained on 250 million image-text pairs.

At test time DALL-E drew 512 samples per caption and kept the one that CLIP (§30.2) scored as the best match for the caption. This reranking spends 512 times the generation compute to buy quality, and a fair comparison must say when a system used it.

Later work showed that nothing special is needed in the generator. A plain Llama language model on the tokens of a good tokenizer (codebook 16,384, downsampling 16) reached an FID of 2.18 on class-conditional ImageNet $256\times256$ at 3.1 billion parameters. The model is LlamaGen (Sun et al. 2024). FID is a distance between the feature statistics of generated and real images, where lower is better, and §34.6 derives it. That model generated $24\times24 = 576$ tokens at 384 pixels and resized the result. Running it in vLLM, a serving engine built for language models, gave a reported 326 to 414 percent speedup. The whole serving stack of language models carries over for free.

The tokenizer sets a floor under all of this. The generator can only write images that the tokenizer's decoder can produce. So tokenizer errors on fine text, small faces, and hands cap the whole system. This is the reconstruction floor of §34.1 in discrete form. For exactly this reason Parti froze its tokenizer's encoder and codebook after training and fine-tuned a larger decoder of about 600M parameters. The codes stayed the same, and the pictures drawn from them got sharper.

Why raster order is slow, and arbitrary

Generation is sequential. Each image token needs one forward pass. A KV cache (§6.5) stores the keys and values of the earlier tokens, so each pass processes only the one new token. At small batch sizes that makes generation memory-bound. The time per step is set by how fast the GPU can read the weights from memory, not by how fast it can multiply.

Batching many images shares each read of the weights among them, which is why serving engines built for language models help so much. At batch size 1, though, the GPU mostly waits. What limits the batch is the cache, and it is large for images.

The order is also arbitrary. Text has a direction, and an image does not. In raster order the top of a face is generated before the model has seen the chin. A token's left neighbor is always available, but its lower neighbor never is. Other fixed orders are possible, but any fixed order makes the model commit to early tokens without the context it will later have. It also makes editing a region in the middle of the image awkward, since only the end of the order can be regenerated given the rest. These two problems, sequential depth and an arbitrary order, drive the rest of the section. An interviewer who asks why anyone left raster order expects both.

Sampling image tokens, and guidance on logits

Sampling an image token is sampling from a softmax over $K$ codes, so the tools of §6.4 apply unchanged. Temperature $\tau$ divides every logit before the softmax, so $\tau < 1$ sharpens the distribution and $\tau > 1$ flattens it. Top-k sampling keeps only the $k$ largest logits, and every token tied at the boundary survives. Image models often sample with temperature 1 and no truncation for the best FID, and with truncation for the most pleasing individual images. That is the familiar trade between diversity and fidelity.

Conditional models need one more tool. Classifier-free guidance (§33.5) makes a model follow its condition more strongly than it learned to. It asks the model for a prediction with the condition and one without, and moves further in the direction the condition pushed. The recipe for autoregressive models is the same as for diffusion. During training the condition is dropped with some probability and replaced by a null condition, an empty caption or a learned embedding. One network then learns both $p_\theta(q_i\mid q_{<i}, \mathbf{c})$ and $p_\theta(q_i\mid q_{<i}, \varnothing)$. At each sampling step the model runs twice on the same prefix, once with each condition, and the two logit vectors are combined.

Write $\boldsymbol\ell_c, \boldsymbol\ell_u \in \R^K$ for the conditional and unconditional logits. Write $\gamma \ge 0$ for the guidance scale in the convention of §33.5, where $\gamma = 0$ samples the unconditional model and $\gamma = 1$ the plain conditional one. Here is what guidance does to three codes.

Proof. A softmax is unchanged by adding the same constant to every logit. Since $\log p_{c,j} = \ell_{c,j} - A_c$ with $A_c = \log\sum_m e^{\ell_{c,m}}$, and likewise for $u$, we have $\ell_{\gamma, j} = (1-\gamma)\ell_{u,j} + \gamma\ell_{c,j} = (1-\gamma)\log p_{u,j} + \gamma\log p_{c,j} + \big((1-\gamma)A_u + \gamma A_c\big)$. The bracket does not depend on $j$, so $\operatorname{softmax}(\boldsymbol\ell_\gamma)_j \propto \exp\big((1-\gamma)\log p_{u,j} + \gamma\log p_{c,j}\big) = p_{u,j}^{1-\gamma}p_{c,j}^{\gamma}$. The last form in the theorem is the same expression regrouped. The temperature identity is linearity. ∎

In plain words, guidance reweights the conditional prediction by the ratio $p_c/p_u$, raised to the power $\gamma - 1$. By Bayes' rule that ratio is $p(\mathbf{c}\mid q_i, q_{<i})/p(\mathbf{c}\mid q_{<i})$, how much more likely the caption becomes once code $q_i$ is written. That is the implicit classifier of §33.5, so codes that make the caption more likely get boosted. With $\gamma = 1$ nothing changes. With $\gamma > 1$ the distribution sharpens toward the codes the caption specifically favors.

Two consequences matter in practice. First, guide logits (or log-probabilities, which differ from logits by a constant), never probabilities. The combination $\mathbf{p}_u + \gamma(\mathbf{p}_c - \mathbf{p}_u)$ with $\gamma > 1$ can produce negative numbers, as the example showed.

Second, the per-token statement does not lift to whole images. The product of the guided per-token distributions is not proportional to $p(\mathbf{q}\mid\mathbf{c})^{\gamma}p(\mathbf{q})^{1-\gamma}$, because the normalizer of each step depends on the prefix. §33.5 found the same gap for diffusion, between what a guided sampler is said to sample and what it does sample. The cell checks the theorem and the first consequence on random logits.

Guidance costs two forward passes per token. They are usually run as one batch of two, with two KV caches, so guidance doubles the work and the cache memory but not the number of sequential steps.

Several systems vary the scale during sampling. Muse (Chang et al. 2023), a masked text-to-image model met later in this section, increases it linearly over the decoding steps. Early tokens are drawn with little guidance and late ones with more, which reduced the loss of diversity. Muse also replaced the null condition by a negative prompt, a caption describing what to avoid, as §33.5 did for diffusion. The released sampler of the next-scale model later in this section (Tian et al. 2024) also ramps the guidance up linearly, scale by scale. Muse's paper and that code both write the rule as $(1+t)\boldsymbol\ell_c - t\boldsymbol\ell_u$, which is our $\gamma = 1 + t$. Always translate a paper's scale into one convention before comparing numbers, using the table of conventions in §33.5.

import numpy as np

def guided_probs(logits_c, logits_u, gamma, k, temperature):
    raise NotImplementedError

def guided_sample(logits_c, logits_u, gamma, k, temperature, rng):
    raise NotImplementedError

Many tokens per step, by masked decoding

The first way out of sequential depth comes from masked language models. BERT (§7.1) is trained to fill in words hidden behind a special mask token. It is bidirectional, meaning every position attends to every other, so it uses the context on both sides of each gap. And it predicts every masked position in one forward pass. Masked decoding turns that into a generator. Start from an image of nothing but mask tokens and predict every token. Keep the predictions the model is most sure of, mask the rest again, and repeat for a small fixed number of steps. The method is MaskGIT (Chang et al. 2022), short for masked generative image Transformer.

In plain words, guess everything, trust only the surest guesses, and guess the rest again with those guesses as context, committing more tokens per step as the picture fills in.

The cosine schedule is concave, meaning its graph bends downward. It stays near 1 for a while and then falls fast, so it commits few tokens while the canvas is nearly empty and many at the end. A convex schedule, such as $1 - \sqrt r$, bends upward and does the opposite. MaskGIT's authors compared concave schedules (cosine, square, cubic, exponential), the linear one, and convex ones (square root, logarithmic). They found the concave family best, with cosine slightly ahead of square on every score. They used cosine in training as well, where $r \sim \mathcal{U}[0, 1)$ makes high mask rates common. The mean training mask rate is $\int_0^1\cos(\pi r/2)\,\mathrm{d}r = 2/\pi \approx 0.64$, far above BERT's fixed 15 percent. The cell prints the tokens committed per step under four schedules.

Why commit few tokens first? Because tokens committed in the same step are drawn independently of each other, and the cost of that independence can be computed exactly. The quantity that measures it is the total correlation of a set of tokens, the information they carry about each other. It is the sum of their separate entropies minus their joint entropy (§0.5 defines entropy as average uncertainty in bits).

Proof. By definition of the KL divergence (§0.5), the left side is $\mathbb{E}\big[\log p(\mathbf{q}_A\mid C) - \sum_i\log p(q_i\mid C)\big]$ with $\mathbf{q}_A$ drawn from the joint. The first expectation is $-H(\mathbf{q}_A\mid C)$. Each $q_i$ under the joint has marginal $p(q_i\mid C)$, so the second is $-\sum_i H(q_i\mid C)$. A KL divergence is zero exactly when the two distributions agree, which here is conditional independence. For the bound, the entropy of discrete variables never decreases when variables are added. By the chain rule of entropy, $H(\mathbf{q}_A\mid C) = H(q_j\mid C) + H(\mathbf{q}_{A\setminus j}\mid q_j, C) \ge H(q_j\mid C)$, so $H(\mathbf{q}_A\mid C) \ge \max_i H(q_i\mid C)$. ∎

In plain words, a parallel step loses exactly the information that the committed tokens carry about each other. That loss is at most the uncertainty of all but the most uncertain of them. Tokens the model is nearly sure of contribute almost nothing, which is the reason to commit the most confident tokens first. One caveat applies. MaskGIT chooses its set $A$ after the draws, by confidence, so the theorem describes the fixed-set version of its step. The selection itself adds a bias, which the toy below will show.

In a natural image the dependence among the remaining tokens is largest when the canvas is empty. Two tokens of an empty canvas both depend on whether this is a dog or a cat, and where it sits. Near the end the dependence is small, since the remaining tokens are mostly local texture given their surroundings. A concave schedule matches that, committing few tokens while their total correlation is high and many once it is low.

The toy has exactly that structure. Each $8\times8$ binary grid is one of four layouts, with the left, right, top, or bottom half on. Then every token is flipped independently with probability 0.05. So the tokens are strongly dependent through the layout and independent given it. The predictor is exact. It computes the posterior over the layouts given the visible tokens, so every error is the decoder's. A sample's flips are its distance to the nearest layout, the number of tokens that differ from it. Real samples average $64\times0.05 = 3.2$ flips. A sample counts as mixed if it is 12 or more flips away from every layout, which real samples essentially never are.

Read the output in two directions. Compare the schedules at one step count first. The cosine schedule produces the fewest mixed samples at every step count, tied with linear at $S = 16$. The convex schedule, which commits most tokens at the start, is worst by far. It mixes layouts in every sample at $S = 2$ and in 86 percent at $S = 4$. Now follow one schedule as the steps grow. More steps mean fewer mixtures. At $S = 8$ cosine decoding is nearly clean, with 0.5 percent mixed samples and 3.3 flips, close to the data's 3.2.

The last column shows the other failure. At $S = 16$ cosine samples have only about 0.5 flips, far fewer than real data. Confidence selection is a biased sampler. A token drawn against its layout has low confidence, so it is re-masked and redrawn until the final steps, and the realistic noise is erased. MaskGIT's released code softens this bias with Gumbel noise, $g = -\log(-\log u)$ for $u$ uniform on $(0, 1)$. Add independent Gumbel noise to each log-probability, take the largest, and the winner is an exact sample from the softmax. This Gumbel-max trick turns picking a maximum into a random draw. MaskGIT adds the noise to the log-confidences before ranking them, scaled by a choice temperature that decays linearly to zero over the steps, 4.5 times $1 - r$ in the released ImageNet configuration. That randomizes which tokens get committed, most strongly in the early steps.

The widget runs the same experiment on a larger grid, 16 by 16 tokens with six layouts, the four halves and two diagonal halves. Real samples there average $256\times0.05 = 12.8$ flips, and a sample counts as mixed at 30 flips or more. Play decodes one sample step by step. The histogram below it shows the flips of 150 samples at the same setting, against the dashed outline of real data.

import numpy as np
import math

def maskgit_train_mask(N, r, rng):
    raise NotImplementedError

def masked_ce(logits, targets, mask):
    raise NotImplementedError

import numpy as np
import math

MASK = -1

def maskgit_decode(predict_logits, N, S, temperature=0.0, rng=None):
    raise NotImplementedError

At real scale MaskGIT used a VQGAN tokenizer with 1,024 codes and downsampling 16, so 256 tokens at $256\times256$. Its bidirectional Transformer has 24 layers of width 768 with 8 heads, and the model has 227M parameters in all. It decoded in 8 steps instead of 256. Its quality beat a raster model six times its size, and its speed advantage grew with resolution, as the box below records.

Masked decoding also edits for free. Start decoding from a partial mask that covers only one region, with everything else visible, and the model regenerates that region given the rest. So inpainting, outpainting, and class-conditional editing need no extra training.

Muse (Chang et al. 2023) took the recipe to text-to-image. It reads the caption with a frozen T5-XXL text encoder of 4.6B parameters and attends to it through cross-attention. A base model writes $16\times16 = 256$ tokens in 24 steps. A second, super-resolution model then writes the $64\times64 = 4{,}096$ tokens of a $512\times512$ image in 8 steps, conditioned on the base tokens. The whole image takes about 1.3 s on one TPUv4 chip. The 3B model scored a zero-shot FID of 7.88 on COCO. Show-o (Xie et al. 2024), met in §31.6, uses the same masked decoding for the image half of one Transformer that also reads and writes text.

Parallelism has a price. MaskGIT gives no exact likelihood, because its training loss is the masked cross-entropy without the weight that would make it a bound (§33.7). And its decoder, with confidence selection, does not sample the distribution the network models.

Coarse to fine, by next-scale prediction

Masked decoding keeps the grid and changes the order. The second way out changes what one step predicts. People draw a sketch before the details. An image pyramid stores an image the same way, as a coarse picture plus corrections at finer and finer scales. Next-scale prediction generates that pyramid. Its unit of generation is a whole token map at one resolution, and "next" means the next resolution. The method is visual autoregressive modeling, VAR (Tian et al. 2024).

That needs a tokenizer whose codes form a pyramid. It is a spatial version of residual quantization (§32.6). Plain residual quantization describes a vector with several codes in turn, each one quantizing what the previous ones missed at the same position. Here stage $k$ quantizes what the earlier, coarser stages missed, at its own resolution $h_k\times h_k$, which grows from stage to stage.

In plain words, describe the image with one token, subtract what that token explains, describe the remainder with four tokens, subtract again, and so on up to full resolution. Every map is a correction to the coarser ones.

The tokens inside one scale are drawn independently, so the theorem on parallel steps above applies to each step. Each step pays the total correlation of its tokens. The difference is the conditioning. Here it is a whole coarser image, which should leave much less dependence among the tokens of the next scale than a sparse, partly filled masked canvas does. VAR's tokenizer at $256\times256$ uses downsampling 16, a shared codebook of 4,096, and the ten scales $(1, 2, 3, 4, 5, 6, 8, 10, 13, 16)$. An image is therefore $1 + 4 + 9 + 16 + 25 + 36 + 64 + 100 + 169 + 256 = 680$ tokens, generated in 10 steps.

import numpy as np

def msrq_encode(f, codebook, scales):
    raise NotImplementedError

def msrq_decode(maps, codebook, n):
    raise NotImplementedError

What each family really costs

VAR's paper states that generating an $n\times n$ token grid costs $O(n^6)$ for an autoregressive Transformer and $O(n^4)$ for VAR. Both statements are true. Comparing them is misleading, because the first counts an autoregressive model without a KV cache, which nobody deploys. The way to settle such claims is to count. For one attention layer, count three things.

• The sequential network calls, which set the latency.
• The tokens pushed through the network's linear layers, which set their FLOPs, about twice the parameter count per token (§6.2).
• The score pairs, the entries of the attention score matrices that a dense implementation computes.

Let $N = n^2$. For VAR, let $T_k = \sum_{j\le k}h_j^2$ be the number of tokens in the first $k$ scales.

In plain words, with caches on both sides, autoregression and VAR do the same order of attention work. VAR in fact does at least $N^2$ pairs against autoregression's $N(N+1)/2$, about twice as many or more. What VAR saves is sequential depth, ten calls instead of 256. Each of its calls also processes many tokens at once, which keeps the GPU busy with arithmetic instead of waiting on memory, the difference between prefill and decode in §6.5. That, not the asymptotic count, is where its measured speed comes from. The cell counts all four families at two grid sizes.

An interviewer who hears "VAR is $O(n^4)$ and autoregression is $O(n^6)$" wants you to ask whether the baseline was cached. The honest answer is that VAR buys latency, not total work.

The widget draws the three counts of the theorem for each family on a log scale. You choose the token grid, turn the KV cache on or off, and give VAR its published scales or scales whose sides shrink by a fixed ratio $a$.

def decode_work(family, n, steps=None, scales=None, kv_cache=True):
    raise NotImplementedError

VAR at real scale

VAR's Transformer is GPT-2-like. The class enters through adaptive layer norm, which lets the condition set the scale and shift of every normalization (§34.2). Queries and keys are normalized to unit length for stability. The width is 64 times the depth, so 1,920 for the depth-30 model, which has 2.0 billion parameters.

On class-conditional ImageNet $256\times256$ that model reached FID 1.92 and an Inception score of 323.1. The Inception score rewards samples that an ImageNet classifier labels confidently and that cover many classes, so higher is better (§34.6). With rejection sampling, which draws extra samples and discards those a classifier scores poorly, it reached 1.73 and 350.2, and a fair comparison must flag that. Both beat the 2.27 of DiT-XL/2, the diffusion Transformer of §34.2 (Peebles and Xie 2023). The autoregressive baseline VAR started from, a 227M-parameter VQGAN Transformer, scored 18.65. VAR's test loss and error rate followed power laws in model size and compute, with correlations near $-0.998$, and the paper received a NeurIPS 2024 best paper award.

Infinity (Han et al. 2024) carried next-scale prediction to $1024\times1024$ text-to-image. Its tokens are strings of bits, made by quantizers that keep only the sign of each latent channel, the lookup-free and binary spherical quantizers of §32.6. So the model predicts each bit with its own two-way classifier instead of one softmax over every possible code. It reported 0.8 s per image and a GenEval score of 0.73 against 0.62 for SD3-Medium. GenEval checks with an object detector whether generated images contain the objects, counts, colors, and positions a prompt asked for (§34.6).

Continuous tokens with a diffusion head

Every model so far pays for discrete tokens twice. The quantizer throws information away, which is the reconstruction floor. And the codebook of $K$ codes must be large to keep that floor low, which makes the softmax over codes harder to learn. So why not let an autoregressive model output continuous vectors directly? The natural candidates are the latents of the continuous autoencoder of latent diffusion, kept near a Gaussian by a KL penalty (§34.1).

The naive answer is to regress the next vector with squared error. Here is why that fails. Suppose that, in some context, the next token is $(1.5, 0)$ or $(-1.5, 0)$ with equal odds. The prediction that minimizes squared error is their mean, $(0, 0)$, a vector that never occurs. In general the minimizer of squared error is the conditional mean, the reason §32.1 gave for blurry autoencoders. When the next token could be one of several quite different vectors, their mean is none of them. A continuous-token model needs a way to sample from $p(\mathbf{x}_i\mid\text{context})$, a conditional distribution over $\R^{d}$. The best general-purpose tool for that in this book is a diffusion model.

In plain words, the big network decides what the next token should be about, and a tiny diffusion model, run once per token, turns that summary into an actual vector, with all the variety a diffusion model can express.

The torch block shows the difference in the smallest possible setting. The "Transformer" is just an embedding of one of four contexts. In each context the next token is one of two opposite points in the plane, each with probability one half. A squared-error head learns the mean, which is the midpoint, and a diffusion head learns both points.

About 98 percent of the diffusion head's samples land within 0.4 of one of the two points, split almost evenly between them. The squared-error head outputs points near the origin, about 1.4 away from any real token. At image scale the same failure is blur.

This idea at ImageNet scale is MAR (Li et al. 2024), short for masked autoregressive model, because it predicts tokens in a random order and several at a time, as masked decoding does. Its tokens are the 16-channel latents of a KL autoencoder with downsampling 16. Its head is a small MLP of residual blocks, conditioned on $\mathbf{z}_i$ through adaptive layer norm, trained with 1,000 noise levels and sampled with 100. The ablations use 3 blocks of width 1,024, and the largest model 12 blocks of width 1,536. Because the head is small, the loss is evaluated at 4 noise levels per token per image at little cost. A temperature $\tau$ scales the noise the sampler injects, the continuous counterpart of the logit temperature.

The paper also showed that the order and the attention pattern matter as much as the loss. Under one setting, about 400M parameters trained for 400 epochs and sampled with guidance, the FIDs on ImageNet $256\times256$ were as follows.

Continuous tokens helped in every row, and most where the model was bidirectional, while committing several tokens per step cost only a little. The largest model, MAR-H with 943M parameters, reached FID 1.55 with 256 decoding steps.

Fluid (Fan et al. 2024) scaled random-order continuous-token autoregression to text-to-image at 10.5B parameters, with a zero-shot COCO FID of 6.16 and a GenEval score of 0.69. It reported that continuous tokens gave much better visual quality than discrete ones. Random order gave much better prompt adherence than raster order. Yet all variants scaled smoothly in validation loss, so the loss alone did not predict which would look better. NextStep-1 (NextStep Team 2025) pairs a 14B autoregressive Transformer with a 157M flow-matching head (§33.6) over continuous tokens, and also edits images.

The price of continuous tokens is the exact likelihood. A diffusion head gives only a bound on $\log p(\mathbf{x}_i\mid\mathbf{z}_i)$, so the model's likelihood is a bound too. Each token also costs a small sampler run, 100 steps of the head in MAR, though the head is tiny next to the Transformer.

Native multimodal image generation

A native multimodal model puts image tokens and text tokens in one sequence and one Transformer, so the same network reads and writes both (§31.6 calls this early fusion). The open designs make the image half in one of three ways. Some write discrete image tokens one by one, exactly the raster recipe of this section, as Chameleon (Chameleon Team 2024) and Emu3 (Wang et al. 2024) do. Janus (Wu et al. 2024) also writes discrete tokens one by one, but reads input images with a separate encoder. Some denoise continuous image patches inside the language model with a diffusion loss, as Transfusion (Zhou et al. 2024) does. Others use masked decoding for images, as Show-o does.

Since 2025 several frontier image generators have been described as built this way. What the closed systems disclose is thin, so here is exactly what their makers state.

OpenAI's addendum to the GPT-4o system card (March 25, 2025) says that, unlike DALL-E, which is a diffusion model, 4o image generation "is an autoregressive model natively embedded within ChatGPT". It adds that the model can take images as input, follow detailed instructions, and render text reliably, and it describes the architecture no further.

Google describes Nano Banana Pro (Gemini 3 Pro Image, November 20, 2025) as built on Gemini 3 Pro. It generates images at up to 4K resolution, blends up to 14 input images while keeping up to 5 people consistent, and marks its outputs with SynthID watermarks. Google also does not describe how the images are produced.

HunyuanImage 3.0 (Tencent Hunyuan 2025) is open and documented. It is built on a mixture-of-experts language model with over 80 billion parameters, about 13 billion of them active per token (64 experts with 8 routed per token, plus a shared MLP). It predicts text autoregressively. It generates images by diffusion on the latents of a VAE with downsampling 16, inside the same Transformer, as Transfusion does. So "autoregressive" in these descriptions can refer to the model as a whole. It says nothing certain about whether the pixels come from token sampling, masked decoding, or diffusion. In an interview, say what is published and stop there.

The case for generation inside a language model is what it brings along. The same network that reads the prompt can reason about it, use world knowledge, and keep context across a conversation. It can also edit images it was given, the in-context editing that §34.7 takes up. The case against is cost. HunyuanImage 3.0 runs 13 billion active parameters at every denoising step and holds all 80 billion in memory. A diffusion Transformer of 2B to 12B parameters spends at most as much per step, often several times less, with far less memory. A raster model with a frontier language model inside pays that model's full cost for every one of 1,024 or more image tokens.

Autoregression and diffusion, side by side

The table puts the five families side by side. It gives the counts of this section at $n = 16$, what each can and cannot do, and the best published FID of each on one benchmark.

No theorem says which family wins, and the frontier systems mix them. What the comparison does settle is what each design buys.

• Raster autoregression buys exact likelihoods and a shared vocabulary with language, at the price of sequential depth.
• Masked decoding buys few steps and free inpainting, at the price of the likelihood and of independence within each step.
• Next-scale prediction buys few steps with a likelihood, at the price of more total work than a cached autoregressive model.
• Continuous tokens buy back the quantizer's loss with a small diffusion model per token, at the price of an exact likelihood.
• Diffusion buys parallel refinement of every position at every step, at the price of many full passes.

Read more

• van den Oord, Kalchbrenner & Kavukcuoglu (2016). Pixel Recurrent Neural Networks, and van den Oord et al. (2016), Conditional Image Generation with PixelCNN Decoders. The masked convolutions, the blind spot, and the gated stacks that fixed it, with the bits-per-dimension results quoted here.
• Ramesh et al. (2021). Zero-Shot Text-to-Image Generation. DALL-E, a 12B Transformer over text and 1,024 dVAE tokens, with CLIP reranking.
• Yu et al. (2022). Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. Parti, an encoder-decoder scaled to 20B parameters on ViT-VQGAN tokens, with guidance on logits.
• Chang et al. (2022). MaskGIT: Masked Generative Image Transformer. Bidirectional masked training, confidence-based parallel decoding, and the schedule ablation.
• Tian et al. (2024). Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. VAR's multi-scale residual tokenizer, next-scale prediction, scaling laws, and the complexity lemmas in its Appendix B.
• Li et al. (2024). Autoregressive Image Generation without Vector Quantization. MAR, the per-token diffusion loss, and the ablation of order, attention, and token type.
• Sun et al. (2024). Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. LlamaGen, a plain Llama on image tokens, with a careful tokenizer study.
• Tencent Hunyuan (2025). HunyuanImage 3.0 Technical Report. An open native multimodal generator, autoregressive for text and diffusion for images inside one MoE Transformer.

Watch next

▶

▶