This section is about two subjects that every interview on vision-language models reaches sooner or later. The first is a different way to build the model. Every design so far in this chapter joins two parts that were trained separately. A vision encoder turns the picture into vectors, and a language model reads those vectors and writes text. A native multimodal model is instead one Transformer trained on text and images together, usually from its first step, and it can write images as well as read them.

The second subject is measurement. A benchmark score can come from looking at the image, from reading the question alone, or from a habit of answering yes. Two numbers show how large these effects can be. On the college-exam questions of the MMMU benchmark, Google's text-only Gemini Pro scored 42.9 percent without seeing any image. Random guessing scores 22.1, and Gemini Pro Vision, shown the images, scored 44.4 (Chen et al. 2024). Another test asks questions of the form "Is there a chair in the image?". Half are about objects that are present and half about plausible objects that are absent. On the hardest version of this test, the original LLaVA said yes 99.1 percent of the time (Li et al. 2023). Neither number measures seeing.

The first half of the section asks this chapter's three questions of native models. Where does the image enter the model? How many tokens does it cost? What is trained, and on what data? We write images as discrete tokens and see why training such a model diverges and what fixes it. Then we keep images continuous with a diffusion loss, and read the evidence on whether native models scale as well as the separately trained kind. The second half builds the tools of honest evaluation. These are the checks that catch answers that ignore the image, the scoring rules that change a benchmark's number, the two standard measures of invented objects, and a decoding rule that removes some of them.

Late fusion joins two trained parts, early fusion trains one network

Start with the design the chapter has built so far. A vision encoder is a Vision Transformer that cuts the image into small square patches and returns one feature vector per patch. It is usually pretrained on its own with CLIP or SigLIP, which teach it to match photos with their captions (§30.2). A small connector maps each feature vector into the language model's input space. A language model pretrained on text then reads the results as if they were words. LLaVA is the standard example (§31.1).

This is called late fusion. Image and text meet late, after each side has learned its own view of the world. Early fusion is the other end. The field uses the phrase in two senses, and papers that seem to disagree are often using different ones.

In plain words, late fusion attaches a seeing module to a reading module, and the two were built separately. Architectural early fusion has no separate seeing module, only one network that learns to see and read together. Early fusion in the training sense has one network that learned both at once, even if a small encoder sits in front of it.

The difference matters as soon as you read a model card. Meta's model card describes Llama 4 as natively multimodal with early fusion (Meta 2025). Yet its images still pass through a vision encoder based on MetaCLIP. Meta's launch post says the encoder was trained separately, together with a frozen Llama model, to adapt it to the language model. So Llama 4 is early fusion in the training sense and not in the architectural one. An interviewer who asks whether Llama 4 is early fusion wants to hear you separate the two.

The table places the designs of this chapter on one line, from the most separate to the most shared.

For a native model the chapter's three questions get new answers. The image enters as tokens of the same sequence, and every layer reads them. It costs whatever its tokenizer produces, and the model pays that budget again when it writes an image. Everything is trained, usually from the first step. That is the source of both the promise and the difficulty. No module was trained for a different goal, but there is also nothing to reuse, and training has new ways to fail.

Images can be written as words from a codebook

The most direct way to put images into a language model is to write them as words. The tool is a vector-quantized autoencoder (§32.5). Its encoder compresses an image into a small grid of vectors. Each vector is then replaced by the nearest entry of a learned list of $K$ vectors, called a codebook. The image becomes a grid of integers between 0 and $K - 1$, the positions of those entries in the list. A decoder turns the integers back into pixels.

Read the grid row by row, which is called raster order, and the image is a sentence in a language with $K$ words. Real image tokenizers add perceptual and adversarial losses so that the decoded images are sharp (§32.6).

To mix those image words with text, give every image code its own token id after the text ids. Then add two marker tokens, written $\texttt{<boi>}$ and $\texttt{<eoi>}$, for the beginning and the end of an image. Here is the whole idea at toy scale.

In plain words, give every image code a word id, write each image into the text as a run of those ids between two markers, and train one language model on the result with the loss it already uses.

The cell builds the toy example and computes an untrained model's loss, split by the modality of each target. Its last line prints a number worth remembering.

Chameleon (Chameleon Team 2024) is this design at full scale. Its image tokenizer resizes and center-crops every image to $512\times512$ pixels and turns it into $32\times32 = 1{,}024$ tokens from a codebook of $K_{\text{img}} = 8{,}192$ entries. Its vocabulary of 65,536 entries includes those 8,192 image codes. One Transformer models every sequence of text and image tokens with the same next-token loss.

Its block is that of Llama 2 (Touvron et al. 2023). Each sublayer first rescales its input with RMSNorm (§5.2). The feed-forward layer is SwiGLU, which multiplies two linear maps of its input, one of them through a smooth gate (§5.3). Positions are rotary (§5.5).

That share is why the plain loss is not neutral. A model trained on many such pairs spends most of its capacity on pixels. Chameleon balances this with its data mixture rather than with loss weights. Its first pretraining stage mixes 2.9 trillion text-only tokens, 1.5 trillion text-image tokens from 1.4 billion pairs, and 400 billion tokens of interleaved web documents. For half of the text-image pairs it puts the image first, so the same pairs teach captioning as well as generation. In practice, log the loss of each modality separately from the first step, because the average hides which modality is learning.

What one discrete image costs

Count the information first. Each of the 1,024 tokens takes one of 8,192 values, which is $\log_2 8{,}192 = 13$ bits. So the whole image carries at most $1{,}024\times13 = 13{,}312$ bits, or 1,664 bytes. The pixels of a $512\times512$ color image are $786{,}432$ bytes. The tokenizer squeezes the picture by a factor of about 473 and must choose what to keep.

Chameleon's authors name what is lost. Their tokenizer reconstructs images that contain a lot of text poorly, which, in their words, upper-bounds the model on OCR-heavy tasks. Compare LLaVA-1.5-7B in §31.1, whose 576 visual tokens are each a vector of 4,096 floating-point numbers.

Then count the compute, with the formulas of §31.1. Prefill is the first forward pass, which reads the whole prompt in parallel before any token is generated. A backbone with $N$ weights spends $2N$ matmul FLOPs per token of prefill, one multiply and one add per weight. The key-value cache keeps every past token's keys and values so that later tokens can attend to them (§6.5). It stores $2Ln_{kv}d_k$ numbers per token, a key and a value of width $n_{kv}d_k$ in each of $L$ layers, where $n_{kv}$ is the number of key-value heads and $d_k$ the head width.

The last line of the box is the price of generating by next-token prediction. Drawing one image takes 1,024 dependent decoding steps. Each step is a full forward pass for one token, the memory-bound regime of §6.5, where reading the weights, not the arithmetic, sets the speed.

Emu3 (Wang et al. 2024) pushes this design as far as it goes. Its tokenizer turns a $512\times512$ image into 4,096 tokens from a codebook of 32,768, an $8\times8$ spatial reduction, with a further $4\times$ reduction in time for video. One 8-billion-parameter Transformer with a vocabulary of 184,622 is trained with next-token prediction alone on text, images, and video. It is then tuned for generation quality with DPO (§17.2). Generating images token by token, and measuring the result, is the subject of autoregressive image generation, which takes these tokenizers further.

import numpy as np

def allowed_next(prefix, n_img, n_text, n_codes):
    V = n_text + n_codes + 2
    return np.ones(V, dtype=bool)          # placeholder: allows everything

def split_loss(logprobs, targets, n_text, n_codes):
    nll = -logprobs[np.arange(len(targets)), targets]

Why training one network on two modalities diverges

When Chameleon's authors scaled this model past 8 billion parameters and 1 trillion tokens with the standard Llama 2 recipe, it diverged. The loss fell normally for a long time and then blew up in the middle or late stages of training. A run could look healthy for its first 20 to 30 percent and still diverge later. They traced the cause to slow growth in the size of the activations. Three small changes to the block fixed it, and two of them appear earlier in the book.

Why do activations grow? The model's output layer produces one score per vocabulary entry, called a logit, and the softmax turns the logits into probabilities. The loss leaves one property of the logits free and pushes another one up. Their overall level is free, because adding the same constant to every logit leaves the softmax unchanged, $\operatorname{softmax}(\mathbf{z}) = \operatorname{softmax}(\mathbf{z} + c\mathbf{1})$. Their scale is pushed upward, because larger logits make sharper predictions, and a sharper prediction lowers the loss when it is right. The same holds for the softmax inside every attention layer.

In a mixed-modal model every weight is shared by text and image tokens, and the two have very different next-token entropies. Each modality pushes the norms of the shared activations up a little to sharpen its own predictions. In the authors' words, the modalities compete. The growth is slow and harmless at first. It turns into divergence once the values leave the range that bf16, the 16-bit training format of §8.5, represents well.

The decisive control was an ablation with image generation removed, which did not diverge. The growth of the last layer's output norm predicted the divergences well in advance, so that norm is worth plotting in any mixed-modal run.

QK-norm is the first fix. It normalizes each query and each key before their dot product, so the attention logits cannot grow with the weights (§8.8 teaches it, and §4.3 shows the same logit growth wrecking an 8-billion-parameter Vision Transformer). Without QK-norm, Chameleon-7B diverged after about 20 percent of an epoch.

z-loss is the second fix. Write $Z$ for the sum of the exponentiated output logits, the normalizer of the output softmax. z-loss adds $10^{-5}\log^2 Z$ to the loss. That small penalty keeps $\log Z$ near zero, which pins down the level of the output logits that the softmax cannot see (§8.8 derives its gradient). QK-norm does not touch the output layer, so both are needed. Chameleon-7B also used dropout of 0.1 after attention and after the feed-forward layers.

The 34-billion-parameter model needed a third change. It moves the normalization from what each sublayer reads to what it writes. Write $\mathbf{x}$ for the block input, $\operatorname{norm}$ for RMSNorm with a learned gain vector $\mathbf{g}$, $\operatorname{attn}$ for the attention sublayer, and $\operatorname{ffn}$ for the SwiGLU feed-forward layer.

In plain words, Llama 2 normalizes the input of each sublayer and adds its raw output to the residual stream, the running sum that every block adds to. Chameleon-34B normalizes the output before adding it. The reordering comes from Swin Transformer V2 (Liu et al. 2022), and its benefit can be stated exactly.

In the Llama 2 order there is no such bound. Scale every feed-forward weight matrix by a factor $\rho$. The update $\operatorname{ffn}(\operatorname{norm}(\mathbf{h}))$ then grows like $\rho^2$ for a ReLU feed-forward layer, and like about $\rho^3$ for SwiGLU, whose gate multiplies two linear maps. The derivation proves both statements.

The cell checks the claim with a ReLU feed-forward layer whose weights have grown by a factor of 1, 4, and 16. The pre-norm update grows exactly like the square of the factor, from 0.72 to 11.5 to 184. The post-norm update stays at root-mean-square 1 with unit gains.

The bound is on each update, not on the residual stream, which still grows by addition. What it removes is the feedback loop in which larger weights make larger updates, which make larger activations.

The Chameleon authors report three further findings. The reordering did not change perplexity before the point where the Llama 2 version diverged. It did not combine well with dropout, so the 34B model was trained without dropout and with z-loss. And they later found that the 7B model also trained stably without dropout once the norms were reordered. QK-norm was essential in both. These are empirical findings from one family of runs, not guarantees. An interviewer who asks how you would stabilize a mixed-modal run wants these three changes, and the output norm as the early warning.

Images can stay continuous, with a diffusion loss

Discrete tokens have two costs of their own. The quantizer throws information away, and writing an image takes 1,024 sequential steps. Diffusion models avoid both. A diffusion model learns to generate by learning to remove noise. In training you add Gaussian noise of a random strength to a clean image and train the network to predict the noise you added (§33.2). To generate, you start from pure noise and remove it over a fixed number of steps, refining every part of the image in parallel at each step.

Transfusion (Zhou et al. 2024) puts both objectives in one Transformer. Text is predicted token by token. Images are kept as continuous vectors and denoised.

Each image is first compressed by a variational autoencoder (VAE), an encoder and decoder trained to reconstruct images through a smaller continuous grid called a latent (latent diffusion uses the same step). In Transfusion's setup a $256\times256$ image becomes a $32\times32\times8$ latent, one 8-dimensional vector per $8\times8$ block of pixels. Each $k\times k$ group of latent vectors, called a patch, becomes one Transformer input of width $d$. A linear layer does the mapping, or the down blocks of a small U-Net, a convolutional network that downsamples and then upsamples. Outputs are mapped back the same way. Text tokens use an ordinary embedding table and output layer, and everything in between is one shared Transformer with a Llama-style block.

In plain words, one network reads one mixed sequence. Where the next element is a word, it predicts the word. Where the next element is an image, it predicts the noise in that image, and the patches of each image may all see each other.

Decoding switches between two modes. In language-model mode the model samples token by token. When it samples $\texttt{<BOI>}$, it appends $n$ patches of pure noise and runs the reverse diffusion process, 250 steps in the paper for a model trained with 1,000 timesteps. Each step overwrites the noisy patches in place, so the model always conditions on the current noise level of the image and never on earlier ones. Then it appends $\texttt{<EOI>}$ and returns to text. The cell builds the attention mask.

The printed mask is for two text tokens, an image of three patches, and one more text token. Each of the three patch rows reads all three patches of its image, and the final text token reads everything before it. Only the $3\times3$ block of the image breaks causality. The authors found that this attention within each image matters most when patches are encoded with a single linear layer. There, replacing it with causal attention raised the FID of generated images from 20.3 to 61.3 (lower is better, defined in the box below). With the U-Net layers the difference was small.

The patch size $k$ sets the token count. A $32\times32$ latent grid becomes $(32/k)^2$ positions, which is 1,024 for $k = 1$, 256 for $k = 2$, 64 for $k = 4$, and 16 for $k = 8$. So a 256-pixel image can cost as few as 16 positions, where Chameleon pays 1,024 for a 512-pixel one (the cost cell above). The larger patches need the U-Net layers to work well. Those layers add 0.27 billion parameters in every configuration, 3.8 percent of the 7B model.

In practice, continuous patches lose less information than a codebook and can cost far fewer positions, and the controlled comparison rewarded both. An interviewer may ask why Transfusion needs full attention inside each image. Diffusion refines all patches of an image at once, so each patch should see every other patch, including those later in raster order, which a causal mask would hide.

Reading and writing can use different encoders

One codebook serves two jobs poorly. To answer a question about an image, a model wants semantic features, vectors that say what is in the picture and drop pixel detail that does not change the meaning. Contrastive encoders such as CLIP and SigLIP learn exactly that (§30.2). To draw an image, a model wants codes that preserve the pixels, which is what a tokenizer trained for reconstruction learns.

Janus (Wu et al. 2024) stops forcing one representation to do both. For understanding it reads images with a SigLIP encoder (SigLIP-Large with 16-pixel patches at 384 pixels, so $24\times24 = 576$ features) and a two-layer MLP adaptor. For generation it writes images as codes of a VQ tokenizer with a codebook of 16,384 that downsamples by 16, again 576 codes at 384 pixels, with its own adaptor and output head. Both paths feed one autoregressive Transformer, a 1.3B DeepSeek-LLM in the first version.

Janus-Pro (Chen et al. 2025) scaled the recipe to 7B parameters and more data. It reports 79.2 on MMBench, a multiple-choice test of visual understanding, and 0.80 on GenEval, which checks whether generated images contain the objects, counts, colors, and positions a prompt asks for. The price is two input paths, and two representations of every image that is used in both directions.

Two other designs split the work differently.

• Mixture-of-Transformers (Liang et al. 2024) shares one self-attention over the whole sequence. Each modality gets its own copy of every other weight, the feed-forward layers, the attention projections, and the norms. In the Chameleon 7B setting it matched the dense model with 55.8 percent of the FLOPs. In the Transfusion setting a 7B model matched the dense model's image quality with a third of the FLOPs.
• BAGEL (Deng et al. 2025) combines these ideas at scale. It is a Mixture-of-Transformers with two experts, one for understanding and one for generation, 7B active of 14B total parameters, initialized from Qwen2.5. It reads images through a SigLIP2 Vision Transformer and writes them as FLUX VAE latents with a rectified-flow loss (§33.6). It was trained on trillions of interleaved tokens.

Weights per modality are a mixture of experts with a fixed router. In §13.1 a learned router picks an expert for each token. Here the token's modality makes the choice. An interviewer may ask why that helps. Each modality gets its own capacity while attention still mixes them, so the compute per token stays the same and quality per FLOP rises.

Any-to-any models

Nothing limits the joint vocabulary to two modalities. Unified-IO 2 (Lu et al. 2023a) is a 7B encoder-decoder model trained from scratch that reads and writes text, images, audio, and robot actions as tokens. To train stably it needed 2D rotary embeddings, QK-norm, and scaled cosine attention in its resampler, the same family of fixes as Chameleon's.

The large chat models moved the same way in 2024 and 2025. OpenAI describes GPT-4o as one network trained end to end on text, audio, images, and video (OpenAI 2024), and since March 2025 it also writes images natively. Google added native image output to Gemini in the same year. Neither company publishes how its images are represented or decoded. So these releases show that unified models are in production, not which of this section's designs they use.

How speech enters and leaves such a model, and what full-duplex conversation requires, is the subject of speech-to-speech models. The same move, adding a new kind of token to the stream, is how vision-language-action models control robots. They append discretized motor commands to the vocabulary of a vision-language model.

Early and late fusion scale at the same rate from scratch

Late fusion has a clear head start. It begins from a language model and a vision encoder that already work, so it needs far less multimodal training. Whether that head start is also a lasting advantage is an empirical question. Shukor et al. (2025) gave the first large answer. They trained 457 models, nearly all from scratch with no pretrained language model or encoder, and fit scaling laws to both families.

Their early-fusion models project raw $14\times14$ pixel patches of $224\times224$ images linearly into the decoder, with causal attention for text and full attention within each image. Their late-fusion models put a Vision Transformer encoder, also trained from scratch, in front of the decoder.

A scaling law here is a power law that predicts validation loss $L$ from training compute $C$ in FLOPs. It also predicts the compute-optimal model, the number of parameters $N^\star$ and training tokens $D^\star$ that reach the lowest loss for a given $C$ (§9.3).

The last bullet of the fact box is why both camps are right. Trained from scratch on the same budget, the encoder is not a privileged module. But a team that already owns a strong language model and a strong encoder starts with their pretraining for free, and that is what late fusion buys.

Frontier labs have moved toward joint training, with details that differ. The Gemini 1.0 report (Gemini Team 2023) describes models that are multimodal from the beginning, trained jointly across text, images, audio, and video, and able to output images as discrete image tokens. Llama 4 (April 5, 2025) is early fusion with a MetaCLIP-based encoder, pretrained on up to 48 images per sequence. Kimi K2.5 (Kimi Team 2026) continues from the Kimi K2 language model with joint pretraining on about 15 trillion mixed vision and text tokens. It also reports a controlled study of when to add vision, and that study points the same way as Shukor et al.

The table compares the four families on what a team deciding between them needs to know.

The architectures of frontier models are mostly unpublished, so what can be said with sources is short. The box lists the documented releases from April 2025 to early 2026. Every score in it is reported by the model's own developers.

Those numbers are only as meaningful as the benchmarks that produce them. The rest of this section is about measurement.

A benchmark score is a protocol applied to a sample

A benchmark score is not a property of the model alone. It is a protocol, the prompt, the answer parser, and the scoring rule, applied to a sample of questions (§22.2). Change any part of the protocol and the number changes. The benchmarks that model reports quote fall into a few families, each with its own scoring rule.

Four dangers run through all of them. The first is that a question can often be answered without the image. The second is that the scoring rule can reward something other than understanding, such as a preference for one answer letter or a habit of saying yes. The third comes with evaluations that ask a judge model to grade open-ended answers (MM-Vet, LLaVA-Bench). They inherit the judge's biases, which §22.4 treats in detail. The fourth is test items that leaked into training data, which inflate everything (§22.3). The next subsections build a check for each of the first two.

Many benchmark questions can be answered without the image

Chen et al. (2024) measured the first danger directly. On six popular multimodal benchmarks they scored each VLM three ways. They scored it with the images, without the images, and through its base language model alone, before any multimodal training. Many questions turned out to be answerable from the text. Gemini Pro's text-only model scored 42.9 on MMMU, against 22.1 for random choice and 44.4 for Gemini Pro Vision with the images.

Some VLMs also scored better without the image than their own base language model could. Nothing in the text-only setting should help them, since the image is gone. The authors attribute it to multimodal training data that contained the benchmark's questions or close relatives of them. Two numbers capture both effects.

In plain words, the multimodal gain is how much the image helps the model that was trained to use it. The leakage is how much better that model does without the image than its text-only parent. It is credit that multimodal training gave the model for text it should not have seen. Leakage is never negative. A model that scores below its base language model once the images are removed has leaked nothing.

The authors then built MMStar from questions that strong language models could not answer without the image and that humans confirmed needed it, 1,500 samples across 6 core capabilities and 18 axes.

MMMU-Pro (Yue et al. 2024) repaired MMMU itself in three steps. It removed questions that text-only models answered correctly too often. It raised the options from four to ten, with GPT-4o proposing distractors, Claude 3.5 filtering them, and two rounds of human review. And it added a vision-only setting in which the question and options appear only inside a screenshot or photo.

The MMMU-Pro score is the mean of the ten-option and vision-only accuracies. It fell 16.8 to 26.9 points below MMMU for the models tested. GPT-4o went from 69.1 on MMMU to $(54.0 + 49.7)/2 = 51.85$. Human experts, estimated from MMMU's own human study, lost only 1.8 to 3.2 points. So most of the models' drop measures shortcuts they had been taking. In practice, before you trust a benchmark gain, rerun it with the images removed.

Scoring rules change the number

ANLS gives partial credit for reading

On documents, a model that reads $12.5 where the answer is $12.50 has read the document and made a formatting slip. A model that answers "twelve" has not read it at all. Exact match scores both 0. The average normalized Levenshtein similarity (ANLS) separates them. It was introduced for scene-text question answering (Biten et al. 2019) and adopted by DocVQA.

ANLS rests on the Levenshtein distance $\mathrm{lev}(a, b)$, the minimum number of single-character insertions, deletions, and substitutions that turn string $a$ into string $b$. Divide it by the length of the longer string and you get the fraction of characters you would have to fix.

In plain words, each answer earns one minus the fraction of characters you would have to fix, but only if you would have to fix fewer than half of them. The best match over the accepted answers counts.

The threshold encodes a judgment. In the authors' words, an answer more than half wrong is a wrong piece of text selected from the image rather than a misread right one. A normalized distance of exactly 0.5 scores 0, because the condition for credit is strict. The cell computes the hand cases.

In practice ANLS is the right score for extractive reading, where small slips should cost a little and wrong fields should cost everything. For fields that feed software, such as dates and amounts, add exact match after normalization, since a slip there is a bug.

def levenshtein(a, b):
    raise NotImplementedError

def anls(predictions, references, threshold=0.5):
    raise NotImplementedError

CircularEval gives credit only for consistent answers

Language models prefer some answer letters over others, whatever the question. §22.2 showed this. It used cyclic reorderings of the options, rotations that move each option through every letter once, to estimate that preference and divide it out. MMBench (Liu et al. 2023) uses the same rotations for a stricter purpose. It asks every question once per rotation of its options and gives credit only when every answer is right.

Take four options. A model that guesses uniformly is right on one pass with probability 1/4. To pass CircularEval it must be right on all four rotations. A model that always says A fails, because the right answer visits every position once and so sits at A in exactly one rotation. The general result is clean.

With four options a uniform guesser earns 25 percent on a single pass and $4^{-4} = 0.39$ percent under CircularEval. A model that always says A earns exactly 0. MMBench reports the effect at real scale. OpenFlamingo v2 fell from 36.7 to 2.6 percent when the evaluation moved from one pass to CircularEval. The cell checks the theorem by simulation, for a pure guesser and for a model that knows 60 percent of the answers.

The one-pass score of the model that knows 60 percent is about 0.70, and 0.10 of that is guessing credit. CircularEval removes almost all of it. The price is up to $n$ times the inference, less in practice because a question can stop at its first wrong pass. A real model's errors are also not independent across passes, so the theorem bounds guessing credit rather than predicting a score.

def circular_eval(ask, question, choices, answer):
    letter = ask(question, list(choices))
    return "ABCDEFGHIJ".index(letter) == answer      # one pass only

def mg_ml(S_v, S_wv, S_t):
    return S_v - S_wv, S_wv - S_t

Object hallucination can be measured in captions and in questions

A VLM hallucinates an object when its output asserts an object that is not in the image. §19.8 treated hallucination in language models, where the cause is a model that must guess about facts it barely saw. In VLMs the main causes are visible in the training pipeline.

• The language model brings strong priors about which objects go together. A dining table suggests chairs.
• Training captions repeat those co-occurrences.
• Synthetic instruction data written from captions can mention details the image does not show (§31.4).
• A low-resolution encoder cannot see small objects at all (§31.5), so the language prior fills the gap.

What contrastive encoders are blind to (§30.6) also bounds what any VLM built on them can check. Two measures dominate. One counts invented objects in free captions, and the other asks about one object at a time.

CHAIR counts invented objects in captions

CHAIR (Rohrbach et al. 2018), short for Caption Hallucination Assessment with Image Relevance, reads generated captions for mentions of the 80 object classes of the COCO dataset. A synonym list maps words to classes, so "man", "woman", and "player" all mean person. Two-word names such as "hot dog" are matched before single words, so that "hot dog" does not count as a dog. An object counts as present if either COCO's segmentation labels or its human reference captions mention it.

The POPE paper (Li et al. 2023) ran CHAIR on the open VLMs of 2023 and found most of them worse than the small captioning models before them. Asked to "Generate a short caption of the image", LLaVA had $\mathrm{CHAIR}_s = 32.7$ and $\mathrm{CHAIR}_i = 10.5$ with captions averaging 64.3 words. The older OSCAR-base had 13.0 and 7.1. Asked instead to "Provide a brief description of the given image", LLaVA rose to 62.7 and 18.8 with 90.7-word captions.

Two prompts with the same meaning moved $\mathrm{CHAIR}_s$ by 30 points. Longer captions mention more objects, and objects mentioned late in a caption are more often invented (Zhou et al. 2023). That instability, and CHAIR's dependence on its parsing rules, motivated POPE. In practice, compare CHAIR only at matched caption length and with the same synonym list.

def caption_objects(caption, synonyms):
    words = caption.lower().split()
    return [synonyms[w] for w in words if w in synonyms]

def chair(captions, gt_objects, synonyms):
    raise NotImplementedError

POPE asks about one object at a time

POPE (Li et al. 2023), short for Polling-based Object Probing Evaluation, turns hallucination into yes-or-no questions of the form "Is there a chair in the image?". In the standard setup it takes 500 COCO validation images with more than three annotated objects and asks 6 questions about each. Three are about objects that are present and three about objects that are absent, so each split has 3,000 balanced questions. The splits differ in how the absent objects are chosen.

• Random samples any absent class.
• Popular takes the most frequent classes in the dataset that are absent from this image.
• Adversarial takes the classes that co-occur most often with the image's own objects, exactly the ones a language prior would suggest.

The scoring treats "yes" as the positive class. Write $a$ for the true positive rate, the fraction of present objects the model says yes to. Write $b$ for the false positive rate, the fraction of absent objects it says yes to. Everything POPE reports comes from the confusion matrix of §0.11, so everything is a function of these two numbers.

In plain words, $F_1$ with yes as the positive class rewards saying yes. Always-yes earns $F_1 = 0.667$ without looking at a single image, while a model that never says yes earns 0. A headline $F_1$ is meaningless without the yes ratio next to it.

The cell recovers $b$ from each reported yes ratio and recomputes every metric.

The widget puts the theorem in motion. A simulated model gives each question an evidence score, drawn from one bell curve when the object is present and another when it is absent. It answers yes when the score clears a threshold, so the shaded part of each curve is its true positive rate $a$ or its false positive rate $b$. The discrimination is the distance between the two curves on random negatives, and the yes bias moves the threshold left. The negatives buttons slide the absent curve toward the present one, as harder negatives do, by amounts chosen for illustration. Below, accuracy and $F_1$ are drawn for every threshold against the yes ratio.

Two practical details decide whether POPE numbers are comparable. The first is the answer parser. The official POPE script keeps the text before the first period and removes commas. It answers "no" if the words "No", "no", or "not" appear, and "yes" otherwise. So an unparseable answer such as "I cannot tell" counts as yes. The lmms-eval harness instead compares the whole lowercased reply with "yes" and "no". There "Yes, there is." matches neither, counts as wrong in accuracy, and drops out of precision and recall. Its yes-ratio metric also counts the ground-truth answers rather than the model's, so it reads about 0.5 on every balanced split whatever the model does. Comparing a number computed one way with a number computed the other is comparing parsers.

The second detail is sampling noise. With 3,000 questions an accuracy near 0.85 has a 95 percent interval of about $\pm1.96\sqrt{0.85\cdot0.15/3000} = \pm0.013$. The six questions about one image are correlated, so the honest interval is wider still (the clustered standard errors of §14.7). Differences of one point between models are not evidence.

def pope_parse(text):
    return "no" if "no" in text.lower() else "yes"

def pope_metrics(answers, labels):
    raise NotImplementedError

Existence is only the first kind of hallucination

POPE asks only whether an object is there. It cannot catch a blue car called red, a cup said to sit on a table it does not touch, three dogs where there are two, or a chart edited to contradict what the model remembers. CHAIR is limited the same way to COCO's 80 class names. When POPE stops separating models, the usual next steps are three. One is a broader benchmark with no judge model (AMBER). One is a test with edited control images (HallusionBench). One is an open-ended set graded by a judge (MMHal-Bench).

A complete hallucination report combines four pieces. A yes-or-no probe with no judge, reported with accuracy and the answer ratio. A rate on generated descriptions, at matched length. A test of the language prior with control pairs. And one judged open-ended set.

Contrastive decoding subtracts the language prior

If hallucinated objects come from the language prior, we can try to subtract the prior at decoding time. Here is the idea in plain words. Run the model twice at every step, once with the real image and once with a copy of the image that has lost its detail. A word that the blurred run likes as much as the real one is coming from the prior, not from the picture, so push it down. A word that only the real image supports gets pushed up.

That is visual contrastive decoding (VCD, Leng et al. 2023). The distorted copy $\tilde I$ of the real image $I$ is, in the paper, the image after many steps of the diffusion forward process (999 steps for POPE, 500 for the other benchmarks), which leaves almost pure Gaussian noise. The same corrupted-image idea appears in the mDPO loss of §31.4, which uses it during training instead of at decoding time.

Write $x$ for the text prompt and the tokens generated so far, $\ell(y\mid I, x)$ for the logit of candidate token $y$, and $p(y\mid I, x)$ for the softmax of these logits over the vocabulary.

The contrasted logits are easier to reason about in two other forms. One writes the rule as a product of the real-image and noised-image distributions, and the other is the guidance rule of diffusion models.

In plain words, each candidate token is scored by how likely it is with the image, times how much more likely the image makes it, raised to the power $\alpha_{\text{cd}}$. A product of experts is this kind of rule, a distribution formed by multiplying several distributions, each raised to a power, and renormalizing. Here the noised-image expert has the negative power $-\alpha_{\text{cd}}$. A word the model would say just as readily without the picture gets no boost, and a word only the picture supports gets one.

The guidance form connects VCD to §33.5. There, classifier-free guidance combines a diffusion model's noise prediction with a condition, $\boldsymbol\epsilon_c$, and without it, $\boldsymbol\epsilon_\varnothing$, as $\boldsymbol\epsilon_\varnothing + \gamma(\boldsymbol\epsilon_c - \boldsymbol\epsilon_\varnothing)$. VCD is the same rule applied to logits, with $\gamma = 1 + \alpha_{\text{cd}}$. A similar head-set constraint appears in §19.8, in a decoding method that contrasts the final layer's predictions with an earlier layer's rather than two inputs.

The plausibility constraint is not optional. The ratio $p(y\mid I, x)/p(y\mid \tilde I, x)$ can be enormous for a token that is improbable under both, and without the cutoff the contrast would promote such tokens. Computing the head from the real-image distribution keeps the choice among tokens the model already considers reasonable.

The cell runs one step on a toy distribution. The text so far is "A dog playing with a", and the image shows a frisbee and no cat.

"cat" had probability 0.163 with the real image, because dogs and cats go together in text. With the noised image it had 0.356, which shows that the preference came from the prior. VCD cuts it to 0.023. "frisbee", which only the image supports, rises from 0.490 to 0.913. "car" and "bench" fall outside the head and get exactly 0.

The widget runs the same step. The distortion slider moves the distorted run from the real image at 0 percent to text alone at 100 percent, and 80 percent is close to the cell. The cutoff buttons set $\beta_{\text{cd}}$ or turn the head off, so every word is kept.

At real scale, VCD with LLaVA-1.5 on POPE's COCO random split raised accuracy from 83.29 to 87.73 and $F_1$ from 81.33 to 87.16, averaged over five runs (Table 1 of Leng et al. 2023). Read the baseline carefully. It is plain sampling from the model's distribution, which hallucinates more than the greedy decoding, always taking the most likely token, that most evaluations use. The LLaVA-1.5 paper (§31.1) reports an $F_1$ of 87.3 for its 7B model on the same split with greedy decoding, about what VCD reaches. The VCD paper's appendix also applies VCD on top of greedy decoding. There the gain is smaller but real, accuracy from 87.10 to 88.49 and $F_1$ from 85.54 to 88.02 (its Table 7). VCD also doubles the cost of every decoding step.

The training-time alternatives were covered in §31.4. One is preference tuning on hallucination-corrected responses (RLHF-V). Another is mDPO's conditional preference term, which forces the model to prefer the true image to a corrupted one. The cheapest fix of all is higher resolution, which LLaVA-1.5 found reduces hallucination. An interviewer who asks how you would reduce hallucination wants one fix at each cause, and a fair baseline for each.

import numpy as np

def vcd_distribution(logits, logits_distorted, alpha, beta):
    z = logits - alpha * logits_distorted
    e = np.exp(z)
    return e / e.sum(-1, keepdims=True)

A tiny early-fusion model in PyTorch

The block below trains a two-layer Transformer with QK-norm on a joint vocabulary of 6 words, 12 image codes, and the two image markers. Each training pair is a caption, "a red square ." or its green or blue version, and a $3\times3$ image whose codes come from the four codes that belong to its color. Half the pairs put the image first, so the same pairs teach captioning and generation. The block reports the loss split by modality against the best achievable values. Then it generates an image from a caption with the decoding constraint of Exercise 31.6f.

Both losses reach their floors within a hundred steps. The text loss settles near 0.11, the cost of not knowing the color word when the caption comes first. The image loss settles near 1.45, the cost of choosing among four codes once the color is known, and among twelve for the first code of an image that comes first. The generated codes all come from the blue range. One model, one loss, and one sampler both describe and draw. That is the whole idea of early fusion with discrete tokens, at the scale of a toy.

To evaluate a real VLM on the benchmarks of this section, use a harness rather than your own scripts, so that prompts, answer parsing, and metrics match published numbers. lmms-eval is one widely used option. Its command-line form looks like this. Task names change between versions, and --tasks list prints the current ones.

python -m lmms_eval \
  --model llava \
  --model_args pretrained=liuhaotian/llava-v1.5-7b \
  --tasks pope,mmstar,docvqa_val,mmmu_val \
  --batch_size 1 \
  --log_samples \
  --output_path ./results

Keep the per-sample logs. They let you recompute POPE's yes ratio, rescore with another parser, and run the blind check of MMStar by evaluating once more with the images removed.

Read more

• Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models. Discrete image tokens in a joint vocabulary, and Section 2.3 on why mixed-modal training diverged and how QK-norm, z-loss, and norm reordering fixed it.
• Zhou et al. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. Next-token and diffusion losses in one Transformer, the intra-image attention mask, and a controlled comparison with Chameleon.
• Shukor et al. (2025). Scaling Laws for Native Multimodal Models. 457 models trained from scratch, the early and late fusion scaling laws, and mixture-of-experts results.
• Wu et al. (2024). Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation. Why one tokenizer serves reading and writing poorly, and the two-encoder design.
• Chen et al. (2024). Are We on the Right Way for Evaluating Large Vision-Language Models? Blind answering and leakage across six benchmarks, the MG and ML metrics, and MMStar.
• Li et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models. CHAIR's instability for VLMs, the co-occurrence analysis, and POPE with its three negative-sampling splits.
• Liu et al. (2023). MMBench: Is Your Multi-modal Model an All-around Player? CircularEval and LLM-based choice extraction, with the measured drop from single-pass scoring.
• Leng et al. (2023). Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. Contrasting with a noised image, the plausibility constraint, and results on POPE and MME.

Watch next

▶

▶

▶