Training a vision-language model means deciding, stage by stage, which of its parts may change and what data they learn from. These choices decide what the finished model can do as much as its architecture does. Getting them wrong can quietly cost the model skills it already had.

A vision-language model (VLM) is a language model that can also read images. The kind this chapter builds has three parts. A pretrained vision encoder turns an image into a grid of vectors. A small new connector maps those vectors into the language model's input space. A pretrained large language model (LLM) reads them alongside the text and writes the answer. A part is frozen during a stage when its weights are held fixed, and trained when they may change.

One controlled experiment shows how much these choices matter. In a study called VILA, Lin et al. (2023) took one vision encoder and the Llama-2-7B language model. They pretrained the pair on 25 million images from web pages in which images and paragraphs alternate. In one run the language model was frozen and only the connector trained. In a second run the language model trained too. Both were then fine-tuned on the same instructions, with the language model unfrozen.

With no examples in the prompt, the two models scored the same, 66.8 on average over two captioning and two question-answering benchmarks. With four solved examples in the prompt, the frozen-LM model fell to 57.6 while the other rose to 68.8. Freezing the language model during pretraining had removed in-context learning, the ability to pick up a task from examples placed in the prompt (§6.8).

A third run kept the language model trainable but swapped the web pages for 25 million images with short web captions. Its score on MMLU, a text-only multiple-choice exam of knowledge in 57 subjects, fell from 46.0 to 28.8 percent. Training on short captions had cost it more than a third of its score on a test that has nothing to do with images. The encoder and the language model were the same in every run. Only what was trained, and on what data, changed.

The earlier sections of this chapter decided where the picture enters the language model and how many vectors it becomes. §31.1 maps each image patch's feature to a vector the size of a word embedding and places those vectors in the input sequence. §31.2 keeps the language model frozen and lets it look at a compressed image through new cross-attention layers. §31.3 compresses the image into 32 vectors with learned queries. This section takes the wiring as given and asks how to train it. At each stage, what is trained, on what data, and what stays frozen?

We will count the memory each stage needs and read the controlled studies on freezing and data. We will prove why a little text-only data protects text skills, and build the data that teaches a model to read, count, and point. Then we adapt preference tuning and reinforcement learning to images, and end with the published recipes side by side.

Each stage solves one problem

The order of training follows from one mismatch. The encoder and the language model are pretrained and know a great deal. The new connector between them starts random and knows nothing. So training proceeds in stages, each fixing one problem.

1. Alignment. Teach the connector to write images in the language model's input space, cheaply, with both large models frozen.
2. Multimodal pretraining. Teach the joined system to use images inside long contexts. This needs documents where images and text alternate, and usually needs the language model itself to train.
3. Instruction tuning. Teach it to follow requests about images in the formats people actually use. This is supervised fine-tuning, or SFT, on conversations with correct answers (§14.1).
4. Preference tuning. Train on pairs of answers where one is marked better, above all to make the model truthful about what is in the picture.
5. Reinforcement learning (RL). Let the model try answers and reward the ones a program can check, which builds step-by-step reasoning.

Two terms recur throughout. An interleaved document is a web page or PDF kept in its original order, with images among paragraphs, as opposed to a single image with its caption. LoRA keeps a weight matrix frozen and trains a small correction added to it, the product of two thin matrices, so the correction has low rank (§14.5).

For models that put image vectors into the token sequence, the first three stages all minimize the same loss. It is the masked next-token loss of §14.3. The model predicts each trained token from everything before it, and the image's slots and the user's tokens are never prediction targets (§31.1). What changes from stage to stage is the data, and the data always come as a mixture of sources.

In plain words, each stage chooses which parameters may move and how often each kind of data is shown. For example, a stage that draws 45 percent of its examples from interleaved pages, 45 percent from captions, and 10 percent from text has $w = (0.45, 0.45, 0.10)$. A batch of 256 examples then holds about 115, 115, and 26 of each. Everything else in this section is evidence about those two choices, the trainable set and the weights $w_k$.

What a stage trains, and what it costs

Freezing is first a memory decision. Recall the count from §10.1. Training one parameter with Adam in mixed precision costs 16 bytes. That is 2 for its weight in bf16 (16-bit floating point), 2 for its bf16 gradient, 4 for a master copy of the weight in fp32 (32-bit floating point), and 8 for Adam's two fp32 moment estimates. A frozen parameter needs only its bf16 weight, 2 bytes, because it has no gradient and no optimizer state.

LoRA sits between the two. Its base weights are frozen at 2 bytes each. Each adapted matrix, mapping $d_{\text{in}}$ inputs to $d_{\text{out}}$ outputs, gains $r(d_{\text{in}} + d_{\text{out}})$ new parameters at rank $r$, and those train at 16 bytes. Here model states means weights, gradients, and optimizer state together, everything except the activations.

Activations are a separate bill, and freezing does not remove it. Activations are the outputs of every layer that the backward pass needs. In stage 1 the projector sits below the whole language model. Its gradient is built from the gradient of the loss with respect to the language model's input, and backpropagation reaches that input only by passing back through every layer above it (§31.1 derives this gradient). So every layer's activations are still stored.

What freezing does save is weight gradients, optimizer state, and about a third of the compute. A forward pass costs about $2N$ FLOPs per token for $N$ parameters, and a full backward pass about $4N$, half for activation gradients and half for weight gradients. A frozen language model skips the weight half, so a step through it costs about $4N$ instead of $6N$. Only a module that no gradient has to cross, such as a frozen encoder at the bottom, can skip storing activations. Its features can even be computed once and saved to disk.

The planner below does the same accounting for several published recipes, stage by stage. Pick a model and a stage, switch a module between frozen, train, and LoRA, and watch the bill move. The bar's full length is what training every parameter would cost.

Here are the answers to check against. Flamingo-80B's trained 13 percent take more than half of its model-state memory, about 163 GB of 304 GB, because each trained parameter costs eight times a frozen one. For Idefics2-8B at rank 64, the language model's line grows from about 17 GB to about 116 GB, roughly sevenfold. The interview version of this is a question you should answer in under a minute. Given a recipe, how much memory do its model states need, and which module dominates?

def lora_count(shapes, r):
    return sum(r * d_in * d_out for d_in, d_out in shapes)

def stage_memory(modules, plan, bytes_frozen=2, bytes_trained=16):
    n_train = sum(n for name, (n, _) in modules.items() if plan.get(name) != "frozen")
    total = bytes_trained * sum(n for n, _ in modules.values())
    return n_train, total

Stage 1, aligning the connector

Alignment trains the new connector alone, on images with captions, so that its outputs become vectors the frozen language model can read. LLaVA-1.5 (Liu et al. 2023b) aligns its MLP projector on 558K captioned images with both large models frozen. It trains for one epoch at a learning rate of $10^{-3}$ and batch size 256. Instruction tuning then runs at $2\times10^{-5}$ with batch size 128. The first learning rate is fifty times higher, and that is safe because only the random projector moves, so nothing pretrained can be damaged. Alignment takes about 6 of the recipe's 26 hours on 8 A100s.

Why not skip alignment and train everything at once? The danger is the first few hundred steps. A random connector produces vectors unrelated to anything the language model has seen. The loss on the caption tokens is large, and its gradient flows into pretrained weights that cost a great deal of compute to learn. Stage 1 keeps that large early gradient away from them. The original LLaVA (Liu et al. 2023a) tested this on ScienceQA, a benchmark of multiple-choice science questions, many with diagrams. Skipping alignment and training directly on ScienceQA lost 5.11 points of accuracy (§31.1).

Other recipes protect the pretrained weights in other ways.

• Skip it, with a large instruction mix. The Prismatic study of VLM design choices (Karamcheti et al. 2024) trained the projector and language model together from the start, on LLaVA-1.5's full instruction mixture. It matched or beat the two-stage recipe on its benchmarks while using 20 to 25 percent less compute.
• Separate learning rates and warmups. A warmup raises a learning rate from zero to its full value over the first steps. Molmo (Deitke et al. 2024) also skips the alignment stage. In its captioning pretraining it gives the connector $2\times10^{-4}$ with a 200-step warmup, and gives the encoder $6\times10^{-6}$ and the language model $2\times10^{-5}$, each with a 2,000-step warmup.
• Train everything, protect the encoder. PaliGemma (Beyer et al. 2024) freezes nothing from the very first multimodal step. It protects its image encoder with a slow linear warmup of that encoder's learning rate. Early in training, while the rest of the model is still unaligned, the gradients reaching the encoder are large and point in poor directions, and the warmup keeps them from degrading it.

All three follow one principle. A new module must start producing sensible outputs before the pretrained modules around it are allowed to move fast. Freezing is the bluntest way to enforce it, and separate learning rates with warmups are the most flexible. In an interview, expect to be asked why alignment uses a learning rate fifty times higher than instruction tuning. The answer is that only the random connector moves, so a large step cannot damage anything pretrained.

Multimodal pretraining, where skills come from

Alignment teaches the connector to describe single images in the language model's input space. It does not teach the system to read several images in one context, to use an image as evidence in a long argument, or to keep its text skills while doing so. Those come from a second, much larger stage, multimodal pretraining. The controlled studies agree on four things it needs.

Unfreeze the language model. In VILA's Table 1, the model whose language model stayed frozen during pretraining matched the unfrozen one with no examples in the prompt (66.8 against 66.8). With four examples it lost 11 points (57.6 against 68.8). The authors measured why. With the language model frozen, image and text embeddings became similar only in the first layers. Unfreezing it aligned them deep in the network too. They call this the deep embedding alignment hypothesis, and it is an empirical finding about their models, not a theorem.

Use interleaved documents, not only captions. A caption is short. It averages about 23 tokens per image in the COYO caption dataset, against 122.5 in the MMC4 dataset of interleaved web pages. A language model trained only on captions drifts toward writing short captions. In VILA's Table 3, pretraining on COYO dropped MMLU from 46.0 to 28.8 percent and made four-shot accuracy worse than zero-shot (50.3 against 51.1). Interleaved MMC4 dropped MMLU only to 40.7 and lifted four-shot accuracy to 70.9.

The authors then cut each MMC4 document into separate image-text pairs, keeping exactly the same text, and the benefit vanished (MMLU 32.4, four-shot 44.5). So the interleaved structure itself, several images among long paragraphs, is what teaches multi-image reading, not merely the longer text. §31.2 showed that a cross-attention model needs such data by design. Here a decoder-only model turns out to need it too.

Mix all three kinds. The MM1 study of multimodal pretraining (McKinzie et al. 2024) varied the mixture of interleaved documents, captions, and text with everything else fixed. It stated four lessons.

• Interleaved data drive few-shot and text-only performance.
• Captions drive zero-shot performance. Raising the caption share lifted zero-shot accuracy from 25.8 to 39.3 percent.
• Text-only data help few-shot and text-only performance.
• Synthetic high-quality captions help few-shot performance.

Its final mixture is 45 percent interleaved documents, 45 percent captioned images, and 10 percent text. It trains every module for about 400B tokens, with up to 16 images per sequence.

Watch the stability. The two pretrained models, the vision encoder and the language model, are often called the backbones. The team behind Idefics2 (Laurençon et al. 2024a) could not train with both backbones unfrozen. The loss diverged even after aggressive learning-rate cuts and gradual unfreezing. LoRA on the backbones, with the new connector trained in full, was stable. It lifted their average score from 60.3 with frozen backbones to 69.5.

Idefics2 then pretrained on about 1.5 billion images and 225 billion text tokens in two phases. The first ran at up to 384 pixels with 70 percent interleaved documents from the OBELICS dataset and 30 percent image-text pairs. The second ran at up to 980 pixels with 45 percent OBELICS documents, 35 percent pairs, and 20 percent PDF pages, which need the resolution to be legible.

At the largest scale these choices merge into continued pretraining of the whole system, and the frontier mixes vision in from the very start. Kimi Team (2026) asked how to spend a fixed budget of vision and text tokens in Kimi K2.5. The usual answer is to add vision late at a high ratio, and they found it worse. Holding the totals fixed, they compared three schedules.

Early, low-ratio mixing won on vision and on text alike. When vision arrived halfway through or later, text performance first dipped and then recovered only slowly. Early mixing avoided the dip. Kimi K2.5 itself continued from the Kimi K2 language model with about 15 trillion more tokens of joint vision and text pretraining.

Why text-only data keeps text skills

VILA and MM1 both found that a small share of text-only data protects text skills. VILA's authors put it as the skills being "temporarily hidden, but not forgotten". Adding 1M text instructions back during SFT recovered the whole MMLU loss, 51.4 percent against 51.2 for the same language model fine-tuned on text alone. Why should a small share be enough?

Take two losses, a text loss $\mathcal{L}_{\text{txt}}$ and a multimodal loss $\mathcal{L}_{\text{mm}}$. Training on a mixture with text weight $w \in [0, 1]$ minimizes $\mathcal{L}_w = w\,\mathcal{L}_{\text{txt}} + (1-w)\,\mathcal{L}_{\text{mm}}$. The question is where the minimum of $\mathcal{L}_w$ lies.

Proof. For every $\theta$, $\mathcal{L}_{\text{txt}}(\theta) \ge m_{\text{txt}}$ and $\mathcal{L}_{\text{mm}}(\theta) \ge m_{\text{mm}}$. With $w > 0$ and $1 - w > 0$, adding the weighted inequalities gives

with equality exactly when both inequalities are equalities. The parameter $\theta^\star$ attains this bound, so the bound is the minimum of $\mathcal{L}_w$. A parameter is therefore a minimizer if and only if it attains the bound, that is, if and only if it minimizes both losses. For $w = 0$ the text term has weight zero, so $\mathcal{L}_0$ cannot tell apart two parameters with equal multimodal loss and different text losses. ∎

In plain words, if one model can be good at both tasks, then any positive share of text data, however small, makes "good at both" the only optimum. The share changes how fast training gets there, not where it ends.

The theorem says where training ends, not what happens on the way. On the way, the text loss rises whenever the multimodal gradient pushes the weights in a direction the text loss cares about. A small text share acts like a spring that pulls the weights back in exactly those directions, with a stiffness proportional to $w$. A weak spring still wins in the end, but slowly. The derivation below makes this precise.

The theorem's assumption is the part to doubt. It holds when the model has the capacity to be good at both tasks. VILA's full recovery suggests that is true for a 7B model and a modest visual curriculum. When capacity is short, no common minimizer exists, and the weight sets a trade-off instead.

The two cells below show both regimes in a toy. The model is a parameter vector $\theta$. The text task and the image task are each a set of linear equations in $\theta$, and the loss is squared error. It is a toy that illustrates the mechanism and nothing more. The evidence that it applies to VLMs is VILA's and MM1's.

Now take away capacity. With 50 parameters for 60 equations, no parameter vector solves both tasks, and the weight sets the compromise.

The first few percent of text data buy the most. Moving $w$ from 0 to 0.05 cuts the text loss from 16.66 to 4.04 at a cost of 0.025 on images. Moving it on from 0.05 to 0.5 buys a further 3.2 of text loss for 1.0 of image loss. The trade-off curve is steepest at the start. That is the logic behind MM1's 10 percent and VILA's joint SFT. A small text share is nearly free when capacity suffices, and the cheapest protection available when it does not.

The data that teaches each skill

After architecture and freezing, data decides what a VLM can do, and each kind of data teaches something specific. The table lists the main kinds, what each is for, and where open recipes got it. OCR, optical character recognition, means reading printed text from an image. Grounding means tying a phrase to a place in the image, as a box or a point. VQA, visual question answering, is the task of answering a short question about an image.

Reading text in images needs data of its own, because captions rarely contain the text an image shows. Idefics2 added PDF pages and rendered text in its second pretraining phase. Its ablation measured DocVQA after a short fine-tune. Raising the resolution from 384 to 768 pixels lifted the score from 22.6 to 42.9, and adding the text-reading data lifted it further to 49.9. The authors saw no loss on other benchmarks.

For the next model, Idefics3, the same team built Docmatix, 2.4 million images and 9.5 million question-answer pairs from 1.3 million PDFs. The pipeline ran OCR on each page and asked a text-only language model to write questions and answers from the transcription. Idefics3 scored 13.7 points higher than Idefics2 on DocVQA, a benchmark of questions about scanned documents (Laurençon et al. 2024b). Pix2Struct (Lee et al. 2022) went further. It pretrained a model to turn screenshots of web pages, with parts masked out, into simplified HTML. That single objective covers OCR, layout, and captioning at once.

Instruction tuning then turns all of this into a chat model. LLaVA-1.5's 665K mixture is the reference point that most open recipes still resemble.

Later recipes kept the idea and grew the mixture. MM1's SFT used about 1.45M examples, including detailed captions and document and chart sets, with text-only instructions to keep text skills. Molmo replaced format prompts with short style tags. It prefixes VQAv2 questions with vqa2:, for example, so the model writes benchmark-style answers only when asked for them. Idefics2 assembled 50 datasets into a collection called The Cauldron and removed any example that overlapped the evaluation sets, a step worth copying (§22.3).

Grounded outputs, boxes and points as text

A language model outputs a bounding box by writing its coordinates as ordinary text tokens in an agreed format. This is the simplest answer, and the one most VLMs use. Nothing about the architecture changes. The model needs only training data in that format.

In plain words, a box is a short sentence of digits, and training teaches the model to say the right digits after the right phrase. Scaling to $[0, 1000)$ gives the same object the same coordinates whatever the image's pixel size. The price is a fixed precision of a thousandth of a side. A predicted box usually counts as correct when its intersection over union (IoU) with the true box is at least 0.5. IoU is the area the two boxes share divided by the area they cover together (§27.6).

The cell turns box annotations into the grounded question-answer strings a data pipeline would emit, in both conventions.

def normalize_box(box, W, H):
    x1, y1, x2, y2 = box
    return tuple(round(1000 * v / W) for v in (x1, y1, x2, y2))

def box_string(label, nbox):
    return f"<ref>{label}</ref><box>{nbox}</box>"

def count_objects(boxes, noun):

Synthetic visual instructions, three ways

When the first open VLMs appeared in 2023, almost no visual instruction data existed, and paying people to write hundreds of thousands of conversations about images is slow and expensive. So every open recipe generates most of it with models. This follows the pattern for text, where a strong model writes instructions and answers for a weaker one, called distillation (§14.2). The three main approaches differ in who sees the image.

A text-only teacher. LLaVA (Liu et al. 2023a) gave a text-only GPT-4 each COCO image's five human captions and its object boxes, plus a few hand-written examples. It asked for three kinds of output, conversations, detailed descriptions, and complex reasoning questions. That produced a 158K set (58K, 23K, and 77K) that both LLaVA and LLaVA-1.5 train on. It is cheap, and its spatial facts are exact because they come from boxes. But the teacher never sees the image. Everything it writes beyond the captions and boxes is an inference that may be false, and the student is trained to say it as if it saw it.

A multimodal teacher. ShareGPT4V (Chen et al. 2023) asked GPT-4V, which does see the image, for 100K detailed captions averaging 942 characters, with prompts adapted to each image source. The authors fine-tuned a captioning model on those and used it to caption 1.2 million more images (826 characters on average). Replacing only the 23K detailed descriptions in LLaVA-1.5's 665K mixture with captions from the 100K set, 3.5 percent of the data, improved LLaVA-1.5 on their benchmarks. Pretraining on the 1.2M captions, with the latter half of the vision encoder's layers unfrozen, helped further.

Dense, accurate captions are among the most effective levers against hallucination in these studies. The price is dependence on a proprietary model. Its terms of use may forbid training competing models, and the student inherits its errors.

People describing images aloud. Molmo's authors (Deitke et al. 2024) wanted open data made without any VLM, in a collection they call PixMo. They asked annotators to describe each image aloud for 60 to 90 seconds, transcribed the speech, and had a text-only language model clean up or merge the transcripts. Talking is faster than typing, and a minimum duration keeps descriptions long. The result was 712K images with captions averaging 196 words, against 11 for COCO's captions.

Other PixMo sets generate data from an exact description, as Docmatix does. For charts, tables, and documents, a language model wrote code that renders an image, 255K images in all. It then wrote 2.3 million question-answer pairs from the code itself, never seeing the pixels. PixMo-Points has 2.3 million pointing annotations on 223K images, which let the model count by pointing at each object in turn.

One principle sits behind all three. The data generator works from a symbolic description of the image (captions, boxes, OCR text, rendering code, a spoken description), and the student sees only pixels. The data are right exactly where the description is complete and the pixels show what it says. If the description mentions something the student cannot see, perhaps because the image will be shrunk to 336 pixels, the student learns to state things it cannot see. Hallucination is exactly that, a confident statement about something the image does not show.

Two observations fit this reading. LLaVA-1.5 found that higher resolution reduced hallucination. RLHF-V (Yu et al. 2023) found the same effect from the other side. Random-crop augmentation can cut away an object that the caption still names, and removing cropping from their augmentation made the model more trustworthy.

Synthetic data also carry the risks §8.4 describes for text. They lose diversity, and when a model trains on its own outputs over several generations, quality can collapse. The defenses are the same. Keep real data in the mixture, filter by quality, and verify a sample of generated answers against the images, with a strong VLM or a person.

Batches that do not mix modalities

One practical detail of instruction tuning saves real compute. Text conversations such as ShareGPT's are often much longer than visual ones. A batch is a rectangle, so every short sequence in it is padded with empty tokens to the length of the longest, and the padding wastes compute. LLaVA-1.5 therefore draws every batch from one modality only. That sped training up by 25 percent without changing the result.

Its released sampler goes one step further, with mega-batches. A mega-batch is one batch for every GPU at once, the number of GPUs times the per-GPU batch size.

1. Shuffle each modality's samples.
2. Cut the shuffled order into mega-batches.
3. Sort each mega-batch by length, so that neighbors have similar lengths.
4. Shuffle the order of the mega-batches.

Sorting inside a mega-batch reduces padding, because the GPUs' batches are then cut from neighbors of similar length. Shuffling outside keeps the order random. Sorting the whole dataset would minimize padding but destroy randomness. Every batch would contain the same neighbors every epoch, and the gradient would sweep from all-short to all-long sequences.

import numpy as np

def modality_length_batches(lengths, is_mm, batch_size, megabatch, rng):
    order = np.argsort(-np.asarray(lengths))         # sorted globally: minimal padding, no randomness
    return [[int(i) for i in order[s:s + batch_size]] for s in range(0, len(order), batch_size)]

In PyTorch

The block below builds a tiny VLM and runs one optimizer step of each of LLaVA's two stages, alignment and instruction tuning. The "encoder" is a frozen random linear layer, the projector is a two-layer MLP, and the language model is a two-layer causal Transformer. Freezing takes two lines, requires_grad_(False) on what must not move and an optimizer built only from what may. Separate learning rates are parameter groups. The prints confirm that stage 1 leaves the language model bit-for-bit unchanged and that the memory bill follows the 16-and-2 rule.

With a real model the same logic is a loop over parameter names. The sketch below uses the Hugging Face LLaVA class and the PEFT library for LoRA, so it is shown rather than run. Freezing by "vision_tower" not in name works whether the language model's head lives under language_model or at the top level, which changed between library versions.

Preference tuning for VLMs

Instruction tuning teaches a model to imitate good answers. It does not teach the model which of its own answers are wrong. The most common wrong answer of a VLM is a hallucination, a confident statement about something the image does not contain. Preference tuning trains on pairs of answers where one is marked better (Chapter 16), so it is the natural fix. Two early systems adapted it to images, and each ran into a weakness of its own.

The first, LLaVA-RLHF (Sun et al. 2023), trained a reward model on human preferences, a network that scores answers. It found the reward model too weak to catch hallucinations until it was also shown the image's human-written captions. The second, RLHF-V (Yu et al. 2023), asked annotators to correct the false parts of an answer instead of ranking whole answers, and got more out of far fewer labels. The box below has their details. Then a later study found a deeper problem with DPO itself, the subject of the rest of this part.

When DPO ignores the image

Plain DPO on image pairs has a deeper problem, which a careful experiment exposed. Recall the DPO loss of §17.2. It raises the probability of the chosen response relative to a frozen reference model and lowers that of the rejected one. With the image $I$ added to the context it reads

where $x$ is the prompt and $y_w$ and $y_l$ are the chosen and rejected responses. The model being trained is $\pi_\theta$, and $\pi_{\text{ref}}$ is the frozen SFT model. The KL coefficient $\beta$ sets how far the model may move from the reference, and $\sigma$ is the logistic function.

The study that proposed the fix below (Wang et al. 2024b) trained one model with this loss and another with the image removed from every preference pair. MMHal-Bench is a set of 96 image questions written to provoke hallucination (Sun et al. 2023). On it, the model that never saw the images did as well, or slightly better. The pairs could be told apart from the text alone, so DPO had learned a preference that did not depend on the image. The authors call this unconditional preference.

Why can that happen? The two responses in a pair share $I$ and $x$, and the loss depends only on the difference of their log-probabilities. Suppose the chosen response is better for reasons visible in the text, such as being more hedged, more fluent, or free of an implausible object. Then the model can raise that difference without consulting the image at all. Nothing in the objective rewards using the picture.

The three terms do different jobs. The first is the usual comparison of two responses, and the other two close shortcuts that plain DPO leaves open.

• $\mathcal{L}_{\text{DPO}}$ is ordinary DPO, two responses under one image.
• $\mathcal{L}_{\text{CoPO}}$, conditional preference, is DPO with the roles swapped, one response under two images. The only way to lower it is to raise $y_w$'s likelihood, relative to the reference, more with the true image than with the corrupted one. Call a policy image-blind if it changes $y_w$'s log-probability by the same amount whatever the image. Then $\hat r_\theta(I_w, x, y_w) = \hat r_\theta(I_l, x, y_w)$, so it pays exactly $\log 2$ for this term, the value at the start of training, however well it separates responses.
• $\mathcal{L}_{\text{AncPO}}$, the anchor, asks the chosen response's implicit reward to exceed $\delta$. With $\delta = 0$ that means $\pi_\theta(y_w\mid I_w, x) > \pi_{\text{ref}}(y_w\mid I_w, x)$. Plain DPO can widen its margin by lowering both responses' probabilities, the likelihood displacement of §17.5, and the anchor penalizes the chosen one for falling.

In plain words, the chosen response under the true image is pushed up three times, by the response comparison, by the image comparison, and by the anchor. The rejected response is pushed down once. The chosen response under the corrupted image is also pushed down once, which teaches the model that $y_w$ is a good answer only when the picture supports it. Each push is weighted by how wrong the model still is on that comparison, $\sigma(-u)$, which goes to zero once the margin is large.

The cell makes the $\log 2$ claim concrete. It scores two policies that separate the responses equally well, one image-blind and one that uses the image.

import numpy as np

def seq_logp(token_logp, mask):
    return (np.asarray(token_logp) * np.asarray(mask)).mean(axis=1)

def mdpo_loss(lw, ll, lwc, rw, rl, rwc, beta=0.1, delta=0.0, copo=True, anchor=True):
    sig = lambda u: 1 / (1 + np.exp(-u))
    u = beta * ((lw - rw) - (ll - rl))

Does mDPO work? The authors trained with the same 10K preference pairs (the LLaVA-Instruct subset of the Silkie dataset), LoRA, and $\beta = 0.1$, and mDPO beat DPO on both models they tried. MMHal-Bench's score is a judge model's rating of each answer, and higher is better. Its hallucination rate is the share of answers judged to contain a hallucination. Object HalBench reports the percent of responses that mention an object not in the image. For both rates, lower is better.

The ablation on Bunny is the informative part. Removing the conditional term dropped the MMHal score from 2.96 to 2.36, removing the anchor dropped it to 2.50, and removing both gave DPO's 2.28. So conditioning on the image does most of the work.

The choice of corruption mattered too. A random unrelated image as $I_l$ scored 2.81, because it is too easy to tell apart from the true one. A crop keeping 20 to 50 percent of the image scored 2.92, against 2.96 for the default. Hard negatives that share most of the image's content teach the most. §31.6 shows how the object hallucination these methods target is measured.

Reinforcement learning with verifiable visual rewards

Reasoning language models are trained with RL on problems whose answers a program can check, called verifiable rewards (§18.3, §18.5). The usual algorithm is GRPO. It samples a group of responses to each problem and scores each response against the average of its group, instead of against a learned value function (§18.4).

Many visual tasks have checkable answers too.

• A geometry problem drawn as a diagram has a number.
• A chart question has a value, compared within a relative tolerance. ChartQA (Masry et al. 2022) counts a number within 5 percent of the true value as correct, following PlotQA (Methani et al. 2020).
• A counting question has an integer.
• A grounding question has a box, and its IoU with the true box can be thresholded.
• A document question has a string to match.
• A GUI task has an action that either reaches the goal state or does not.

GLM-4.1V-Thinking (GLM-V Team 2025) is a detailed public account of doing this at scale. Two lessons stand out.

The verifier is the product. The same answer can be right in one domain and wrong in another. "43" and "43.0" are equal in a math problem and different in OCR. So the authors built a separate verifier for each domain and recommend unit-testing each against that domain's outputs. A weak verifier anywhere could destabilize the whole run, and they report a concrete case. After more than 500 iterations with a carelessly designed verifier, the model answered a counting question with "a correct number between 0 and 10", which some LLM-based reward models accepted. They also made the model wrap its final answer in special box tokens, so the verifier never has to guess which number in a long chain of thought is the answer.

Sampling should follow difficulty. Their curriculum sampling labels each problem's difficulty offline, by pass@k with existing models (the chance that at least one of $k$ samples is right), and online, from the current model's rollouts. It then down-samples problems that are already easy or still hopeless. Both kinds give a group of identical rewards, and identical rewards give every response a GRPO advantage of zero, so the model learns nothing from them.

Visual RL has one failure of its own. A policy can earn rewards by answering from language priors without looking, since many benchmark questions can be answered from the text alone. RL then strengthens exactly the habit we want to remove. One defense is in the data. Before training, ask the model each question with the image removed, and drop or down-weight the questions it still answers correctly. §31.6 shows how the MMStar benchmark applies the same test to evaluation.

The exercise builds the verifier for four kinds of answer, and the blind filter on top of it. Its tests include the failures described above. One is GLM-4.1V's hedged count. Another compares numbers as floats where the task is to read text exactly. A third reads a box from the middle of the reasoning instead of from its end.

import re

def parse_box(text):
    m = re.search(r"<box>\((\d+),(\d+)\),\((\d+),(\d+)\)</box>", text)
    return tuple(int(v) for v in m.groups()) if m else None

def iou(a, b):
    inter = (min(a[2], b[2]) - max(a[0], b[0])) * (min(a[3], b[3]) - max(a[1], b[1]))

Thinking with images

Text reasoning lets a model spend more tokens on a hard problem (§18.7). A VLM that has encoded an image once, at 336 or even 1,000 pixels, cannot spend more on looking, however long it thinks. The detail it missed was never encoded. OpenAI's announcement of o3 and o4-mini in April 2025 introduced the phrase "thinking with images". It describes models whose chain of thought includes operations on the image itself, such as cropping, zooming, and rotating. Each result is fed back into the context.

DeepEyes (Zheng et al. 2025b) shows how to train this behavior with RL alone, starting from Qwen2.5-VL-7B with no supervised reasoning traces. During a rollout, one sampled attempt at a problem (a trajectory, as in §15.1), the model may write a bounding box. The environment crops that region from the full-resolution image and appends it to the context as new image tokens, up to 6 times per rollout. The appended tokens are observations, not actions, so the loss masks them out, exactly as tool outputs are masked in agent training. For a rollout $\tau$, the reward is

In plain words, the model earns a reward for a correct final answer, a reward for the right output format, and a bonus for using the zoom tool at least once that is paid only when the answer is right. They trained with GRPO for 80 iterations of 256 prompts with 16 rollouts each, without a KL penalty. On V*, a benchmark of small objects in very large images, accuracy rose from 71.2 to 90.1.

The interview follow-up is cost, since every zoom adds hundreds of image tokens to the context. The deeper remedy for fine detail is to encode it in the first place, the subject of §31.5.

A recipe at real scale

The table puts the published recipes side by side. Each row says what each stage trains, on how much data, and the one choice worth remembering.

Read down the second column and a trend appears. Early recipes froze as much as possible, because data were scarce and compute was short. Later ones train nearly everything, gently, on far more data. The frontier trains vision and language together from the start of pretraining, the native multimodal design of §31.6.

The freezing choices, side by side, are what an interviewer expects you to weigh.

Read more

• Lin et al. (2023). VILA: On Pre-training for Visual Language Models. The cleanest controlled study of freezing the LLM, interleaved versus caption data, and joint SFT with text-only data (Tables 1, 3, and 4).
• McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training. Data lessons 1 to 4 and the 45, 45, 10 mixture, with ablations at small scale carried to 30B.
• Liu et al. (2023b). Improved Baselines with Visual Instruction Tuning. LLaVA-1.5's 665K mixture, response-format prompts, and the data-efficiency study in Section 5.
• Laurençon et al. (2024a). What matters when building vision-language models? Idefics2. Six findings from controlled experiments, including LoRA as the fix for divergence when unfreezing the backbones.
• Beyer et al. (2024). PaliGemma: A versatile 3B VLM for transfer. A recipe that freezes nothing, with the encoder warmup and the freezing ablations of Section 5.4.
• Deitke et al. (2024). Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models. Spoken dense captions, data generated from code, pointing, and training without a connector-only stage.
• Li et al. (2024). LLaVA-OneVision: Easy Visual Task Transfer. One open recipe for single images, multiple images, and video, with the stage-by-stage configuration of Table 1 and the task-transfer examples of Section 7.
• Wang et al. (2024b). mDPO: Conditional Preference Optimization for Multimodal Large Language Models. The image-free DPO experiment, the conditional and anchored terms, and the ablations of Tables 2 to 4.
• Zheng et al. (2025b). DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning. Training zoom-in actions inside the chain of thought with RL, and the tool-reward ablation.
• Kimi Team (2026). Kimi K2.5: Visual Agentic Intelligence. Joint text and vision pretraining under a fixed token budget, zero-vision SFT, and visual RL that also lifts text benchmarks.

Watch next

▶

▶