How a language model learns to see. Every way of wiring a vision encoder into an LLM, how such models are trained, how they read documents and video at high resolution, how natively multimodal models differ, and how to measure what they actually see, taken to interview depth.

Chapter 31 · Section 31.1 · 65 min

Connecting a vision encoder to an LLM

A vision-language model, or VLM, is a language model that can read images. You give it a photo and a question, and it writes the answer in words. This chapter builds such models, and this section builds the simplest design that works, which is still the design of most open VLMs. It matters now because chat assistants are expected to read screenshots, charts, and photos as readily as text.

Two earlier pieces make the design possible. A Vision Transformer cuts an image into small squares called patches and turns each one into a vector (§27.1). CLIP and SigLIP train such an encoder so that a photo's vectors point the same way as its caption's (§30.1, §30.4). §30.6 showed what those encoders still miss, word order, counts, and positions, and this chapter's models inherit those blind spots.

The design has three parts. An image encoder turns the photo into a few hundred vectors, one for each small square of the picture. A small network called the projector translates each vector into the language model's input space. The translated vectors go into the prompt where words would go, and the language model reads them as it reads text. The encoder is frozen, which means training never changes its weights.

The design matters because it is cheap to build. One team started from a 13-billion-parameter chat model and an image encoder trained to match photos with their captions. On one machine with eight A100 GPUs, they trained for about 6 hours on image-caption pairs and then about 20 hours on instructions, 1.2 million public examples in all (Liu et al. 2023b). The result, LLaVA-1.5, set the state of the art on 11 benchmarks when it appeared. In the first of its two training stages, only about 31 million weights changed, a quarter of a percent of the system.

It is also expensive to run, in a way that is easy to underestimate. Each image becomes hundreds of tokens, and every one of them passes through every layer of the language model. This section explains both facts, why the training is so cheap and what each image costs.

Why does so little training suffice? Neither half has to learn to see or to speak. The encoder already turns a picture into vectors that carry its meaning, and the language model already turns vectors into fluent answers. What is missing is a translation between their two vector spaces. Translations between learned spaces are often close to linear. §1.10 found exactly that for word vectors, where one rotation, learned from a small dictionary, carries English word vectors close to their Spanish translations.

This chapter asks three questions of every design that lets a language model see. Where does visual information enter the model? How many visual vectors does the model process, and what do they cost? What is trained, on what data, and what stays frozen? This section gives the simplest answers. The image enters as extra tokens at the input, one per square of the image. A frozen encoder and a small projector produce them, and at first only the projector is trained.

We will follow one photo through the whole system, then define the model, count what an image costs, build the projector, and write the code that inserts image vectors into a batch of text. Then we derive the gradient that trains the projector through a frozen language model, and trace LLaVA from its first version to LLaVA-1.5. We end on what this design cannot do, which the rest of the chapter fixes.

A language model reads vectors, not words

A language model never sees words. Its first step replaces each token id $x_t$ by row $x_t$ of an embedding table $\mathbf{E} \in \R^{V\times d}$ , where $V$ is the vocabulary size and $d$ the model width (§6.2). From then on, every layer acts on vectors of $d$ numbers. Nothing in the layers checks that an input vector is a row of $\mathbf{E}$ .

Prompt tuning shows that a frozen model can use this freedom (Lester et al. 2021, §14.6). It trains a few vectors that are not rows of $\mathbf{E}$ and puts them in front of the prompt. The model's own weights stay fixed, yet it follows those vectors as if they were a written instruction.

So here is the plan. Write the image as a sequence of vectors in $\R^d$ , the space where word embeddings live, and place them in the input. The model will attend to them like any other tokens.

Where would such vectors come from? CLIP's image encoder, recalled above, is trained contrastively (Radford et al. 2021, §30.1). From 400 million image-caption pairs, it learned to place each image's embedding near the embedding of its own caption and far from the other captions. Its features therefore already encode what a caption would say about the picture. They live in the encoder's space, not the language model's. But if both spaces organize the same meanings, a simple map should carry one into the other. Here is that idea at its smallest, with numbers you can check by hand.

✏️ A projector in two dimensions, by hand

Let the language model's input space be $\R^2$ , with three word embeddings, "cat" $= (1, 0)$ , "dog" $= (0.8, 0.6)$ , and "car" $= (-0.6, 0.8)$ . Suppose the encoder's space is the same picture turned by 90 degrees and stretched by 2. A photo whose meaning is $\mathbf{e}$ then gets the feature $\mathbf{z} = \mathbf{e}\mathbf{M}$ with $\mathbf{M} = 2\begin{pmatrix}0 & 1\\ -1 & 0\end{pmatrix}$ , so a cat photo has $\mathbf{z} = (0, 2)$ .

Given one photo of each word, least squares finds the projector $\mathbf{W}_{\text{p}}$ that makes $\mathbf{z}\mathbf{W}_{\text{p}}$ as close as possible to $\mathbf{e}$ , summed over the three pairs. Here the fit is exact, $\mathbf{W}_{\text{p}} = \mathbf{M}^{-1} = \tfrac12\begin{pmatrix}0 & -1\\ 1 & 0\end{pmatrix}$ .

A new cat photo, a little off at $\mathbf{z} = (0.2, 1.9)$ , maps to $(0.2, 1.9)\,\mathbf{W}_{\text{p}} = (0.95, -0.1)$ . Its cosine with "cat" is $0.95/\sqrt{0.95^2 + 0.1^2} = 0.995$ , against $0.733$ for "dog" and $-0.680$ for "car". The language model would read it as a cat.

Python · cell 1

A real image is not one vector but hundreds, one per square, and the language model reads all of them. The sequence below is what LLaVA-1.5's chat format shows the model after a short system prompt. The placeholder <image> is replaced by 576 projected vectors.

USER: <image>\nWhat is in the picture? ASSISTANT: A cat asleep on a sofa.</s>
       ^^^^^^^
       576 vectors in R^4096, one per 14 x 14 patch of a 336 x 336 photo

Two facts carry this from a toy to a working system. First, contrastive training makes an encoder's features linearly predictive of what captions say. That is why a linear probe, a linear classifier trained on frozen CLIP features, already classifies ImageNet well (§30.3). Second, a language model's input space already holds one vector for the meaning of every token, and the model accepts vectors that are not among them, as prompt tuning showed.

Neither fact guarantees that a good linear map between the two spaces exists. That one works as well as it does is an empirical finding, and the first version of LLaVA tested it with a single matrix. An interviewer who asks why a linear projector works at all wants these two facts and that caveat.

One photo, end to end

Before any general definition, follow one real input through LLaVA-1.5-7B, the 7-billion-parameter version, from pixels to answer. The photo is a 640 by 480 picture of a cat asleep on a sofa, and the user asks "What is in the picture?" The box lists each stage, what happens there, and the shape of what comes out. The rest of the section explains why each stage looks the way it does.

✏️ One photo through LLaVA-1.5-7B

stage	what happens	output
1. pixels	The photo is padded to a 640 by 640 square with the encoder's mean pixel color, then resized to 336 by 336. Each color channel is normalized with the encoder's mean and standard deviation.	$336\times336\times3$ , 338,688 numbers
2. patches	The image is cut into a 24 by 24 grid of 14 by 14 squares called patches, 576 of them, each holding $14\cdot14\cdot3 = 588$ numbers. One linear map turns each patch into 1,024 numbers. A learned class token, an extra vector that will gather a summary of the whole image, goes in front. A learned position vector is added to each of the 577 rows.	$577\times1{,}024$
3. encoder	The 24 Transformer layers of the frozen CLIP ViT-L/14 encoder mix the rows, at about 0.38 TFLOPs (trillion floating-point operations). LLaVA-1.5 reads the output of layer 23, the second-to-last, and drops the class token.	$\mathbf{Z}$ , $576\times1{,}024$
4. projector	Each row passes through a two-layer network, 1,024 to 4,096 numbers, a GELU (a smooth version of the ReLU nonlinearity), then 4,096 to 4,096. It has 21 million weights and costs 24 GFLOPs for the whole image.	$\mathbf{X}_v$ , $576\times4{,}096$
5. splice	The chat prompt is tokenized. The `<image>` placeholder is replaced by the 576 rows of $\mathbf{X}_v$ , and each text token by its row of $\mathbf{E}$ . Positions count $0, 1, 2, \dots$ through all of it.	576 image rows and a few dozen text rows, each of width 4,096
6. prefill	The 32 layers of the language model read every row in one forward pass, called prefill. The image's rows cost about 7.5 TFLOPs, at least 24 ms on an A100, whose fp16 peak is 312 TFLOP/s. Each layer stores every row's key and value so that later steps need not recompute them. This KV cache takes 512 KiB per row, 288 MiB for the image.	the KV cache, and the scores for the first answer token
7. decode	The model writes "A cat asleep on a sofa." one token at a time. Each step runs one new row through the 32 layers, which reads all 13.5 GB of fp16 weights and the whole cache, at least 7 ms per token at the A100's 2 TB/s of memory bandwidth. The answer ends when the model writes the end token `</s>`.	the answer

In plain words, the photo becomes 576 vectors in the language model's input space, and the model reads them before the question exactly as it would read 576 words.

Put the stages on a clock. With a prompt of about 50 text tokens, the image is about 92 percent of the language model's prefill work, 7.5 of 8.1 TFLOPs. Prefill takes at least 26 ms on one A100, and the answer of about ten tokens takes at least 70 ms more. So the user waits mostly on decoding, while the image decides how much memory the request holds.

The widget plays the same seven stages. Press Play, or step through them one at a time.

The cell runs the same stages at toy size. The image is 42 by 42 pixels with 14-pixel patches, the encoder has width 8, and the language model has width 6. The weights are random, so any answer would be meaningless, but every shape is the real one scaled down.

Python · cell 2

Two numbers from the walkthrough drive the rest of the chapter. The image is 576 tokens, about ten times the text, so it dominates the cost of the request. And the only part trained from scratch is the projector of stage 4, 21 million weights next to 7 billion.

The late-fusion model, defined

We now fix notation for the whole chapter. An image is $I \in \R^{H\times W\times C}$ with height $H$ , width $W$ , and $C$ color channels. It is written in italics so it never meets the identity matrix $\mathbf{I}$ . A text prompt is $x$ , and a response is $y = (y_1, \dots, y_{|y|})$ , as in Chapter 14.

📐 Late-fusion vision-language model

A late-fusion vision-language model has three parts.

A vision encoder $f_{\text{v}}$ maps the image to one feature vector per patch, $\mathbf{Z} = f_{\text{v}}(I) \in \R^{N_p\times d_{\text{vis}}}$ . It is usually a Vision Transformer (ViT), a Transformer that reads an image as a sequence of patches (§27.1). It is usually trained contrastively, like CLIP or like SigLIP, a CLIP variant that scores each image-caption pair with its own sigmoid (§30.4). Here $N_p$ is the number of patches and $d_{\text{vis}}$ the encoder's width (1,024 for CLIP ViT-L/14). We write $d_{\text{vis}}$ , never $d_v$ , which is the width of an attention value.
A connector $g$ maps the features into the language model's input space, $\mathbf{X}_v = g(\mathbf{Z}) \in \R^{n_v\times d}$ , giving $n_v$ visual tokens of width $d$ . For a projector, $n_v = N_p$ .
A decoder-only language model with embedding table $\mathbf{E}$ reads a sequence in which the image's placeholder is replaced by the rows of $\mathbf{X}_v$ , and each text token by its embedding.

The model defines

p_\theta(y \mid I, x) = \prod_{t=1}^{|y|} p_\theta\big(y_t \mid \mathbf{X}_v, x, y_{\lt t}\big), \qquad \mathbf{X}_v = g\big(f_{\text{v}}(I)\big),

where $\theta$ collects every trainable parameter. Image positions are inputs only. The model never predicts them, and the loss never scores them.

In plain words, encode the image into patch vectors, translate each one into the language model's word space, put them in the prompt, and let the language model write the answer exactly as it writes any continuation.

Late fusion means that image and text are first processed by separately pretrained parts and meet only afterwards. Its opposite, early fusion, trains one Transformer on image and text tokens from the first step of pretraining, which is the subject of §31.6.

Here are LLaVA's answers to the chapter's three questions. Visual information enters as extra input tokens. There are $n_v = N_p$ of them per image, and every one passes through every layer. In the first stage only $g$ is trained, and the encoder stays frozen throughout.

In practice, late fusion lets you upgrade the encoder, the connector, or the language model on its own, and reuse the pretraining of each side as it is. An interviewer will ask what that modularity costs. The answer is the token count, which the next two parts of this section measure, and the information the frozen encoder throws away, which the end of this section proves no connector can recover.

Which features, and how many

Recall how a ViT reads an image (§27.1). It cuts the image into non-overlapping $P\times P$ squares called patches. It maps each patch to a vector with one linear layer, and adds a learned position embedding to each. An $H\times W$ image therefore gives

N_p = \Big\lfloor \frac{H}{P} \Big\rfloor \Big\lfloor \frac{W}{P} \Big\rfloor

patch tokens. The floor is there because the patch step is a convolution whose $P\times P$ kernel moves $P$ pixels at a time. Pixels left over at the right and bottom edges, fewer than $P$ of them, fit no whole patch and are dropped.

A CLIP ViT also puts one extra learned vector in front of the patches, the class token (CLS). Through attention it gathers a summary of the whole image. Its final state, after a LayerNorm and a linear map, is the image embedding that the contrastive loss trains (§30.2). SigLIP (Zhai et al. 2023) has no class token. It pools all the final patch tokens with one attention layer instead (§30.4).

Python · cell 3

✏️ Counting LLaVA-1.5's image tokens

CLIP ViT-L/14 at 336 pixels has $P = 14$ , so each side holds $336/14 = 24$ patches and $N_p = 24\times 24 = 576$ . With the class token, each layer outputs 577 vectors. LLaVA-1.5 reads the second-to-last layer and drops the class token, so $\mathbf{Z} \in \R^{576\times 1024}$ . After the projector, $\mathbf{X}_v \in \R^{576\times 4096}$ for the 7B model.

The first LLaVA used the same encoder at 224 pixels, a $16\times16$ grid, 256 tokens. SigLIP SO400M/14 at 384 pixels gives $\lfloor 384/14\rfloor = 27$ patches per side, with the last 6 pixels dropped, so 729 tokens and no class token. A $448\times 672$ image at $P = 14$ would be a $32\times 48$ grid, 1,536 tokens, if the encoder could take it.

CLIP's learned position embeddings were trained for one grid. They can be interpolated to other grids (§27.4), but LLaVA-1.5 does not do that. It pads every photo to a square and resizes it to $336\times336$ first.

Which layer. LLaVA's authors compared the features after CLIP's last layer with those before it, on ScienceQA, a benchmark of multiple-choice science questions with pictures (Lu et al. 2022). The last layer scored 89.96 percent and the layer before it 90.92 (Liu et al. 2023a, Table 8). Their hypothesis is that the last layer concentrates on the global, abstract properties the contrastive loss rewards. The layer before it keeps more local detail.

Why drop the class token. It is one summary of the whole image. §30.6 showed that a single vector per image struggles to hold counts and positions. The class token is also computed from the patch tokens, so dropping it loses little, and the patch tokens keep the locality the language model needs to say what is where. In the Hugging Face implementation, these two choices are the configuration fields vision_feature_layer = -2 and vision_feature_select_strategy = "default", which removes the class token.

Why a contrastive encoder at all. Its features are already organized by what captions say, so a cheap map can align them with the language model. DINOv2 is an encoder trained without any text (Oquab et al. 2023). It learns by making the features of different crops of one image agree (§28.4). It keeps finer geometry, but its features are organized by visual similarity, not by words. We return to that trade-off at the end of the section, because some strong systems use both.

What one image costs

Every visual token is a full member of the sequence. It passes through every layer, and every later token attends to it. Its key and value also sit in the KV cache of every layer, the store of past keys and values that lets a model generate without recomputing them (§6.5). So we can price an image with counting rules the book already has.

Two phases of generation matter. Prefill is the single forward pass over the whole prompt, image included, that happens before the first answer token. Decode is the loop that follows. It writes one token per step and reads the whole cache at every step.

Here are the symbols. The language model has $L$ layers of width $d$ , $h$ query heads of width $d_k = d/h$ , and $n_{kv}$ key-value heads. It has $N$ weights in its $L$ blocks, not counting the embedding table and output head, as in §5.6. The image contributes $n_v$ tokens to a prompt with $n_t$ text tokens, $T = n_v + n_t$ in all. Each cached number takes $b$ bytes, 2 for fp16. A FLOP is one floating-point multiply or add.

Tokens. A projector gives $n_v = N_p$ . Some models merge each $\kappa\times\kappa$ block of neighboring patches into one token. They may also add $n_{\text{special}}$ marker tokens, such as the start and end tokens of Qwen2-VL (Wang et al. 2024a). That gives $n_v = N_p/\kappa^2 + n_{\text{special}}$ . Merging belongs to §31.5. We include it here only so the formula covers real models.

Matmul FLOPs. A token's trip through the blocks uses each of the $N$ weights in one multiply and one add, so it costs $2N$ FLOPs (§5.6). The image's tokens add $2Nn_v$ . They skip the embedding lookup, and in inference the output head runs only at the last position, so neither is counted.

Attention FLOPs. Attention compares each query with every earlier key and averages the matching values. That work grows with the square of the sequence length. The image's share is

F_{\text{attn}} = 2Ld\big[(n_v + n_t)^2 - n_t^2\big] = 2Ld\,(n_v^2 + 2n_vn_t).

∂ Where the attention count comes from

In one layer, query $t$ of a causal prefill scores $t$ keys and averages $t$ values. Over all $h$ heads of width $d_k$ , scoring costs $2td$ FLOPs and averaging another $2td$ , so $4td$ together. Summing over $t = 1, \dots, T$ gives $4d\,T(T+1)/2 \approx 2dT^2$ per layer, and $2LdT^2$ for all $L$ layers. The image's share is what the full prompt costs minus what the text alone would cost, $2Ld\big[(n_v + n_t)^2 - n_t^2\big]$ , and expanding the square gives $2Ld\,(n_v^2 + 2n_vn_t)$ .

KV cache. Each token stores a key and a value of width $n_{kv}d_k$ in each of the $L$ layers. By the formula of §6.5 the image holds $2Ln_{kv}d_kn_vb$ bytes.

The box prices LLaVA-1.5-7B. Its language model is Vicuna-7B, a chat model fine-tuned from LLaMA on conversations that users shared from ChatGPT (Chiang et al. 2023). LLaVA-1.5 uses Vicuna's version 1.5, which is built on Llama 2.

📌 What one image costs

n_v = \frac{N_p}{\kappa^2} + n_{\text{special}}, \qquad F_{\text{prefill}} \approx 2Nn_v + 2Ld\,(n_v^2 + 2n_vn_t), \qquad M_{\text{KV}}^{\text{img}} = 2Ln_{kv}d_k\,n_v\,b .

Vicuna-7B has $L = 32$ , $d = 4{,}096$ , $n_{kv} = h = 32$ , $d_k = 128$ , and $N = 6.48$ billion weights in the blocks. One $336\times336$ image is 576 tokens. It costs about $7.5\times10^{12}$ FLOPs of matmuls in prefill, plus 0.15 TFLOPs of attention next to a 200-token prompt. It holds $2\cdot32\cdot32\cdot128\cdot576\cdot2 = 301{,}989{,}888$ bytes, 288 MiB, of fp16 KV cache. Quick estimates often count all 6.74 billion parameters instead, which gives 7.8 TFLOPs.

In plain words, an image costs what 576 text tokens would cost, about a page of prose.

The cell computes $N$ from each model's configuration, then prices one image and eight images for three language models. The third, a Llama-3-8B-style model, uses grouped-query attention, in which groups of query heads share one key-value head, here $n_{kv} = 8$ (§12.5). That makes its cache four times smaller than Vicuna-7B's. Its FLOPs differ only because its blocks hold a different number of weights.

Python · cell 4

The output shows something the single-image numbers hide. With one image, attention is 2 percent of the prefill. With eight, it is about 9 percent, because its cost grows with the square of the token count.

✏️ LLaVA-1.5-13B, counted

Vicuna-13B has $L = 40$ , $d = 5{,}120$ , $h = n_{kv} = 40$ , $d_k = 128$ , and a SwiGLU feed-forward layer of width 13,824. Each block holds

2d^2 + 2d\,n_{kv}d_k + 3d\,d_{\text{ff}} = 52{,}428{,}800 + 52{,}428{,}800 + 212{,}336{,}640 = 317{,}194{,}240

weights, so $N = 40\times 317{,}194{,}240 = 12{,}687{,}769{,}600$ .

One image of 576 tokens adds $2N\cdot576 = 1.46\times10^{13}$ matmul FLOPs. Next to a 200-token prompt it adds another $2\cdot40\cdot5{,}120\,(776^2 - 200^2) = 2.30\times10^{11}$ FLOPs of attention, under 2 percent of the matmuls.

Its cache is $2\cdot40\cdot40\cdot128\cdot2 = 819{,}200$ bytes per token. That is 471,859,200 bytes, or 450 MiB, for the image, against 156 MiB for the 200 text tokens.

The decode phase pays again. Every new token reads the whole cache. A batch of 32 requests that each carry one photo reads $32\times450$ MiB, about 15 GB, per decoding step. That is more than half the 26 GB of fp16 weights the step also reads (§20.1 turns bytes per step into latency).

The widget prices a request of up to 16 images and a 200-token text prompt. Choose the encoder, a merge factor, the number of images, and the language model.

In practice these three numbers decide whether a VLM can be served. The token count sets the context you need. The FLOPs set the time to the first answer token. The cache sets how many requests fit on a GPU at once. An interviewer who asks "what does an image cost?" wants all three, each with its formula and one real number.

Warm-up

What an image costs

Implement image_cost(H, W, P, kappa, n_special, lm, bytes_per=2). The image has height H and width W and is cut into patches of side P. Each kappa by kappa block of patches is merged into one token, and n_special marker tokens are added. The language model is the dict lm, with keys "L", "d", "h", "n_kv", and "N" (the weights in the blocks).

Return (n_v, kv_bytes, prefill_flops). The token count is $n_v = \lfloor H/P\rfloor\lfloor W/P\rfloor/\kappa^2 + n_{\text{special}}$ . The cache is $M_{\text{KV}}^{\text{img}} = 2Ln_{kv}d_kn_v b$ with $d_k = d/h$ . The prefill cost is the matmul FLOPs $2Nn_v$ . If either side of the patch grid is not divisible by $\kappa$ , raise ValueError, because a $\kappa\times\kappa$ merge needs whole blocks (real models resize the image first so that this never happens). Return all three counts as exact Python integers, not floats.

def image_cost(H, W, P, kappa, n_special, lm, bytes_per=2):
    n_v = (H / P) ** 2 / kappa
    kv_bytes = lm["L"] * lm["h"] * (lm["d"] // lm["h"]) * n_v * bytes_per
    prefill_flops = 2 * lm["N"] * n_v
    return n_v, kv_bytes, prefill_flops

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

The projector

The connector of LLaVA is a projector, a map applied to each patch vector on its own. As everywhere in the book since §4.2, a weight matrix has its input dimension first and multiplies row vectors from the right, as in $\mathbf{Z}\mathbf{W}$ .

📐 Linear and MLP projectors

Let $\mathbf{Z} \in \R^{N_p\times d_{\text{vis}}}$ hold one encoder feature per row, and let $\mathbf{1}$ be the column of $N_p$ ones, so $\mathbf{1}\mathbf{b}^\top$ adds $\mathbf{b}$ to every row.

The linear projector of LLaVA is $\mathbf{X}_v = \mathbf{Z}\mathbf{W}_{\text{p}} + \mathbf{1}\mathbf{b}^\top$ with $\mathbf{W}_{\text{p}} \in \R^{d_{\text{vis}}\times d}$ and $\mathbf{b} \in \R^d$ . It has $d_{\text{vis}}d + d$ parameters. LLaVA's paper writes it without the bias, but its code uses a linear layer, which has one.
The MLP projector of LLaVA-1.5 is

\mathbf{X}_v = \operatorname{GELU}\big(\mathbf{Z}\mathbf{W}_1 + \mathbf{1}\mathbf{b}_1^\top\big)\mathbf{W}_2 + \mathbf{1}\mathbf{b}_2^\top, \qquad \mathbf{W}_1 \in \R^{d_{\text{vis}}\times d},\ \mathbf{W}_2 \in \R^{d\times d},\ \mathbf{b}_1, \mathbf{b}_2 \in \R^d,

with $d_{\text{vis}}d + d + d^2 + d$ parameters. $\mathbf{W}_1$ and $\mathbf{W}_2$ are the two weights of a feed-forward layer, the same symbols as in §5.3, here with hidden width $d$ . GELU acts on each number separately, $\operatorname{GELU}(u) = u\,\Phi(u) = \tfrac{u}{2}\big(1 + \operatorname{erf}(u/\sqrt2)\big)$ , where $\Phi$ is the standard normal CDF.

Both act on each row separately, so $n_v = N_p$ and the order of the patches is kept.

In plain words, a projector is a per-patch translator. The linear one can rotate, stretch, and shift the encoder's space onto the language model's. The MLP can also bend it.

Because a projector treats patches independently, it adds no information about which patch is next to which. The spatial layout reaches the language model in two ways only. One is what the encoder's own position embeddings already mixed into each feature. The other is the order of the patches in the sequence, row by row. §31.5 gives the language model 2D positions directly.

Did the extra layer matter? LLaVA-1.5's paper adds one change at a time and measures each. Replacing the linear projector by the MLP, with everything else fixed, raised three scores (Liu et al. 2023b, Table 2).

MME, a suite of yes-or-no perception and reasoning questions (Fu et al. 2023), went from 1,323.8 to 1,355.2.
GQA, compositional questions about real photos (Hudson and Manning 2019), went from 46.8 to 47.3.
MM-Vet, open-ended questions graded by GPT-4 (Yu et al. 2023), went from 26.3 to 27.8.

The gain is real but small next to the gains from better data and higher resolution in the same table. That is the first hint of a pattern this chapter keeps finding. The connector matters less than what the language model is shown.

Core

The LLaVA-1.5 projector

Implement two functions.

mlp_projector(Z, W1, b1, W2, b2) takes features Z of shape (..., N_p, d_vis), with any number of leading batch dimensions. It returns $\operatorname{GELU}(\mathbf{Z}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2$ with the exact GELU $u\,\Phi(u)$ .
projector_params(d_vis, d, kind="mlp") counts the parameters of the MLP projector. With kind="linear" it counts those of a linear projector with bias.

scipy.special.erf is available. In PyTorch, nn.GELU() computes the exact form by default, and nn.GELU(approximate="tanh") computes the approximation that GPT-2 used.

import numpy as np

def mlp_projector(Z, W1, b1, W2, b2):
    return np.maximum(Z @ W1, 0) @ W2

def projector_params(d_vis, d, kind="mlp"):
    return d_vis * d + d * d

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Splicing image features into the sequence

Now the mechanics. A tokenized prompt holds a placeholder for each image, and the language model must receive the rows of $\mathbf{X}_v$ in its place. Everything else about the sequence must move with them, the labels, the attention mask, and the positions. We call this step the splice.

📐 The spliced sequence

Let the text tokens be $x_1, \dots, x_n$ , with one image between $x_a$ and $x_{a+1}$ . The language model's input is

\mathbf{X} = \begin{bmatrix} \mathbf{E}_{x_1} \\ \vdots \\ \mathbf{E}_{x_a} \\ \mathbf{X}_v \\ \mathbf{E}_{x_{a+1}} \\ \vdots \\ \mathbf{E}_{x_n} \end{bmatrix} \in \R^{(n + n_v)\times d},

where $\mathbf{E}_{x}$ is row $x$ of the embedding table. Several images are inserted the same way, in order.

LLaVA's language model uses 1D rotary position embeddings, RoPE, which rotate each query and key by an angle set by its position (§5.5). Its positions run $0, 1, \dots, n + n_v - 1$ through the image slots without restarting. Models with multimodal RoPE give image tokens 2D positions and advance the text by the image's larger side instead (§31.5).

The training loss is the masked cross-entropy of §14.3,

\mathcal{L} = -\frac{1}{|\mathcal{C}|}\sum_{t\in\mathcal{C}}\log p_\theta\big(s_{t+1}\mid s_{\le t}\big),

where $s$ is the spliced sequence and $\mathcal{C}$ is the set of positions whose target $s_{t+1}$ is a trained token. In chat data, a trained token is a token of an assistant reply. Every target that is an image slot, a system or user token, or padding has weight 0. In code each such slot gets the label $-100$ , the value PyTorch's cross-entropy skips by default.

In plain words, cut the text at the placeholder, push the image's vectors in, shift everything after it, and train only on the answer.

One consequence of causal attention matters for every data format. In a causal model each token attends only to the tokens before it, so a visual token influences only the positions after it. If an image came after every trained target, no loss term would depend on it, and its gradient would be exactly zero. That is why every format puts the image before the answer it should inform, usually at the start of the user's turn.

The widget shows a batch of two chat examples before and after the splice. After it, each slot shows its position, its mask bit, and whether its label is trained or $-100$ .

Interview

Splice image features into a batch

This function is at the heart of every VLM codebase. It makes a good interview problem because every line can hide a silent bug. Implement splice_batch(input_ids, attention_mask, labels, image_features, E, image_token_id, padding_side="right").

input_ids and attention_mask have shape (B, T). The real tokens are those with mask 1. Each image appears as one image_token_id among the real tokens.
labels has shape (B, T) and is aligned with input_ids (unshifted, as in Hugging Face). It is already $-100$ on prompt tokens, and it may hold anything at image tokens and pads.
image_features is a list of arrays of shape (n_v_i, d), one per image, in reading order across the batch (row 0 from left to right, then row 1, and so on). The counts may differ from image to image.
E is the embedding table, of shape (V, d).

Return (inputs_embeds, mask, labels, position_ids) with shapes (B, T', d), (B, T'), (B, T'), and (B, T'), where $T'$ is the length of the longest expanded row. The mask, labels, and positions are integer arrays.

Image slots get their feature rows and the label $-100$ .
Pads get zero embeddings, mask 0, label $-100$ , and position 0.
Positions count $0, 1, 2, \dots$ over each row's real slots.
padding_side says where each output row's pads go, at the end ("right") or at the start ("left").
If the number of image tokens differs from the number of feature arrays, or a feature array has the wrong width, raise ValueError.
Do not modify the inputs.

import numpy as np

def splice_batch(input_ids, attention_mask, labels, image_features, E, image_token_id, padding_side="right"):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

Training LLaVA in two stages

LLaVA trains in two stages, and the encoder stays frozen in both (Liu et al. 2023a).

Stage 1, feature alignment. Take 595K image-caption pairs filtered from CC3M, a set of 3.3 million web images with their alt-text captions (Sharma et al. 2018). Turn each pair into a one-turn conversation, a randomly chosen request to describe the image briefly, followed by the original caption as the answer. Freeze the encoder and the language model, and train only the projector to make the caption likely. The paper calls the result a visual tokenizer for the frozen language model. The name fits, because the projector turns an image into input vectors the language model can read, as a tokenizer does for text.

Stage 2, instruction tuning. Unfreeze the language model and train it together with the projector on 158K multimodal instructions. They are 58K conversations, 23K detailed descriptions, and 77K complex reasoning questions. A text-only GPT-4 wrote them from the human captions and object boxes of images in COCO, a dataset of everyday photos (Lin et al. 2014). How such data is made, and what it does to a model, belongs to §31.4.

Stage 1 trains a few million parameters through a 13-billion-parameter network that does not change. What exactly does it compute? Write $\mathbf{G}$ for the gradient of the loss with respect to a layer's output, as in §0.2.

∴ Gradient of a projector through a frozen language model

Let the loss $\mathcal{L}$ be any differentiable function of the spliced input $\mathbf{X}$ , computed by a language model whose weights are held fixed. Let the visual tokens be $\mathbf{X}_v = \mathbf{Z}\mathbf{W}_{\text{p}}$ with $\mathbf{Z}$ fixed. Write $\mathbf{G}_v = \partial\mathcal{L}/\partial\mathbf{X}_v \in \R^{n_v\times d}$ for the rows of $\partial\mathcal{L}/\partial\mathbf{X}$ at the image slots, the gradient that backpropagation delivers to the visual inputs. Then

\frac{\partial\mathcal{L}}{\partial\mathbf{W}_{\text{p}}} = \mathbf{Z}^\top\mathbf{G}_v \in \R^{d_{\text{vis}}\times d}.

For a batch of images, the gradients of the images add, $\sum_i \mathbf{Z}_i^\top\mathbf{G}_{v,i}$ . For the mean loss over a batch of $B$ , each $\mathbf{G}_{v,i}$ already carries its factor $1/B$ .

Proof. $\mathcal{L}$ depends on $\mathbf{W}_{\text{p}}$ only through $\mathbf{X}_v$ , and $\mathbf{X}_v = \mathbf{Z}\mathbf{W}_{\text{p}}$ is a linear layer with input $\mathbf{Z}$ . For a linear layer $\mathbf{Y} = \mathbf{X}\mathbf{W}$ , the gradient of a scalar with respect to the weight is $\mathbf{X}^\top\mathbf{G}$ , where $\mathbf{G}$ is the gradient with respect to the output (§0.2). With $\mathbf{X} = \mathbf{Z}$ and $\mathbf{G} = \mathbf{G}_v$ this is the claim. For several images, $\mathbf{W}_{\text{p}}$ is shared, so by the multivariate chain rule the contributions of all the places it is used add. ∎

∂ The same gradient for the MLP projector

Apply the linear-layer rule twice. Let $\mathbf{U} = \mathbf{Z}\mathbf{W}_1 + \mathbf{1}\mathbf{b}_1^\top$ and $\mathbf{H} = \operatorname{GELU}(\mathbf{U})$ , so $\mathbf{X}_v = \mathbf{H}\mathbf{W}_2 + \mathbf{1}\mathbf{b}_2^\top$ . The second layer gives

\frac{\partial\mathcal{L}}{\partial\mathbf{W}_2} = \mathbf{H}^\top\mathbf{G}_v, \qquad \frac{\partial\mathcal{L}}{\partial\mathbf{b}_2} = \mathbf{G}_v^\top\mathbf{1}.

The gradient reaching $\mathbf{H}$ is $\mathbf{G}_v\mathbf{W}_2^\top$ . GELU acts on each number separately, so the gradient reaching $\mathbf{U}$ is $\mathbf{G}_U = (\mathbf{G}_v\mathbf{W}_2^\top)\odot\operatorname{GELU}'(\mathbf{U})$ , where $\odot$ multiplies entry by entry. The first layer then gives

\frac{\partial\mathcal{L}}{\partial\mathbf{W}_1} = \mathbf{Z}^\top\mathbf{G}_U, \qquad \frac{\partial\mathcal{L}}{\partial\mathbf{b}_1} = \mathbf{G}_U^\top\mathbf{1}.

In plain words, backpropagation through the frozen language model tells each visual token which direction in the model's input space would have made the caption more likely. The projector's weights then move so that its outputs go that way.

The theorem also says what stage 1 costs. To get $\mathbf{G}_v$ , backpropagation must still run through every layer of the language model. So all its activations are still stored, and every linear layer still computes the gradient with respect to its input, $\mathbf{G}\mathbf{W}^\top$ . What stage 1 skips are the weight gradients $\mathbf{X}^\top\mathbf{G}$ , which cost as much as the forward pass, and all optimizer state for the frozen weights.

So training compute per token falls from about $6N$ FLOPs to about $4N$ . Memory falls further. The frozen 7B model needs only its 13.5 GB of bf16 weights, plus the activations. Full training with mixed-precision Adam would take about 108 GB, at 16 bytes per parameter for the bf16 weights and gradients, fp32 master weights, and two Adam moments (§10.1). The frozen encoder needs no backward pass at all, so its features can even be computed once and stored.

The PyTorch block near the end of the section checks this theorem against autograd. It also checks the causal claim of the previous section, that an image placed after every trained target receives exactly zero gradient.

📌 LLaVA and LLaVA-1.5 in numbers

LLaVA (Liu et al. 2023a) uses CLIP ViT-L/14 at 224 pixels (256 visual tokens), a linear projector, and Vicuna-13B (and 7B). Stage 1 trains the projector only, on 595K filtered CC3M pairs, for 1 epoch at learning rate $2\times10^{-3}$ and batch 128, in under 4 hours on 8 A100s. Stage 2 trains the projector and language model on 158K instructions, for 3 epochs at learning rate $2\times10^{-5}$ and batch 32, in under 10 hours. The encoder is frozen in both. LLaVA scored 85.1 percent relative to GPT-4 on its synthetic instruction benchmark, and 90.92 percent on ScienceQA after fine-tuning (92.53 with GPT-4 as a judge between answers).

LLaVA-1.5 (Liu et al. 2023b) uses CLIP ViT-L/14 at 336 pixels (576 tokens), a two-layer GELU MLP, and Vicuna-v1.5 7B and 13B. Stage 1 uses 558K image-caption pairs, learning rate $10^{-3}$ , and batch 256. Stage 2 uses a 665K mix of LLaVA's 158K, text-only conversations users shared from ChatGPT, and academic VQA, OCR, and region datasets, at learning rate $2\times10^{-5}$ and batch 128. Each stage runs one epoch, in about 6 and 20 hours on 8 A100s, 1.2M public samples in all.

LLaVA-1.5 changed three things besides the projector. It raised the resolution from 224 to 336 pixels. It added academic task data, so the model learned to read text in images and to localize regions. These are short-answer visual questions (VQAv2, GQA, OKVQA, A-OKVQA), questions and captions that need the text in the image (OCR-VQA, TextCaps), and descriptions of marked regions (RefCOCO, Visual Genome).

And it fixed a problem those datasets create. Their answers are one or two words, and a model trained on them turns terse everywhere. So LLaVA-1.5 appended a response-format prompt to every short-answer question, "Answer the question using a single word or phrase." The model learned when to be brief. In Table 2 of the paper, adding VQAv2 alone raised MME from 809.6 to 1,197.0, and adding the format prompt raised it further to 1,323.8.

Is stage 1 necessary? The evidence points both ways. In LLaVA's ablation on ScienceQA, skipping the alignment stage lost 5.11 points (85.81 against 90.92). Prismatic, a later controlled study of VLM design choices (Karamcheti et al. 2024), trained on the full LLaVA-1.5 mix and found the opposite. Skipping stage 1, and training the projector and language model together from the start, matched or beat two stages and saved 20 to 25 percent of the compute.

Both are empirical findings from different settings. With a small fine-tuning set, the alignment stage carries much of the knowledge. With a large and varied instruction mix, it adds little. An interviewer who asks "do you need stage 1?" wants exactly that answer, with both numbers.

Interview

Train a projector through a frozen head

A toy frozen language model reads the mean of the visual tokens, $\mathbf{h} = \frac{1}{n_v}\mathbf{1}^\top\mathbf{Z}\mathbf{W}_{\text{p}} \in \R^d$ . It scores a one-word caption with the logits $\mathbf{E}\mathbf{h} \in \R^V$ , where the word table $\mathbf{E} \in \R^{V\times d}$ is frozen. The batch Z has shape (B, n_v, d_vis), and the caption words y have shape (B,).

Implement loss_and_grad(Z, W_p, E, y). It returns the mean cross-entropy and $\partial\mathcal{L}/\partial\mathbf{W}_{\text{p}}$ , computed by the theorem as $\sum_b\mathbf{Z}_b^\top\mathbf{G}_{v,b}$ .
Implement train(Z, y, E, steps=300, lr=0.5, seed=0). It starts from W_p = 0.01 * rng.normal(size=(d_vis, d)) with rng = np.random.default_rng(seed), runs steps steps of plain gradient descent, $\mathbf{W}_{\text{p}} \leftarrow \mathbf{W}_{\text{p}} - \eta\,\partial\mathcal{L}/\partial\mathbf{W}_{\text{p}}$ with $\eta$ = lr, and returns (W_p, losses). Here losses[k] is the loss at the start of step k.

The setup provides make_caption_data().

import numpy as np

def loss_and_grad(Z, W_p, E, y):
    raise NotImplementedError

def train(Z, y, E, steps=300, lr=0.5, seed=0):
    raise NotImplementedError

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

How simple can the connector be

Why did LLaVA-1.5's MLP help only a little, and when would it help a lot? A small simulation separates the two cases.

We make paired "image" features and "text" embeddings. The text side is a linear function of the image side, plus a squared term that no linear map can predict, plus noise. We fit two connectors. One is a linear projector fit in closed form by ridge regression, least squares with a penalty on the size of the weights. The other is a small GELU MLP trained by gradient descent.

We score both by image-to-text retrieval on held-out pairs. The score, recall@1, is the fraction of images whose nearest text by cosine is their own (§30.1). Squared error stands in for the language model's loss here, so the numbers illustrate the mechanism and nothing more.

Python · cell 5

The linear map finds the right caption among 500 for 33 percent of held-out images, against 0.2 percent by chance. The MLP finds it for 86 percent. The squared term is uncorrelated with every linear function of a Gaussian input, so no matrix can capture it, while the MLP's bend can.

The lesson cuts both ways. A connector needs capacity for whatever part of the relation between the two spaces is not linear. When that part is small, a linear map already gets most of the way. LLaVA-1.5's modest MLP gain suggests that for CLIP features and a Llama-family model the part is small.

Core

A closed-form connector

You have paired image features $\mathbf{Z} \in \R^{n\times d_{\text{vis}}}$ and text embeddings $\mathbf{T} \in \R^{n\times d}$ . The ridge projector $\mathbf{W}^\star = (\mathbf{Z}^\top\mathbf{Z} + \lambda\mathbf{I})^{-1}\mathbf{Z}^\top\mathbf{T}$ minimizes $\lVert\mathbf{Z}\mathbf{W} - \mathbf{T}\rVert_F^2 + \lambda\lVert\mathbf{W}\rVert_F^2$ , where $\lVert\cdot\rVert_F$ is the square root of the sum of squared entries. Implement three functions.

fit_ridge(Z, T, lam) returns $\mathbf{W}^\star$ . Compute it with np.linalg.solve rather than by forming the inverse, which is slower and less accurate.
recall_at_1(Z, T, W) returns the fraction of rows $i$ for which text $i$ has the highest cosine with $\mathbf{z}_i\mathbf{W}$ among all rows of T.
evaluate(Z, T, lam, n_train) fits on the first n_train pairs and returns (W, recall). The recall is measured on the remaining pairs only, searching among their texts.

The setup provides make_pairs(), whose text vectors have lengths that vary by a factor of 20.

import numpy as np

def fit_ridge(Z, T, lam):
    return np.linalg.lstsq(Z, T, rcond=None)[0]

def recall_at_1(Z, T, W):
    return float(np.mean(np.argmax((Z @ W) @ T.T, axis=1) == np.arange(len(Z))))

Tests, hints, verified solutions, and AI help, with Pro at $7.99 a month.

In PyTorch

The block below builds a tiny frozen language model and an MLP projector like LLaVA-1.5's. It splices with masked_scatter, as the Hugging Face implementation does, so input_ids already holds one placeholder per visual token. It trains nothing. It checks three claims of this section.

The frozen model receives no weight gradients, while the projector does.
The projector's gradient is $\sum_b\mathbf{Z}_b^\top\mathbf{G}_{v,b}$ , exactly as the theorem says.
An image placed after every trained target receives a gradient of exactly zero.

🔥 In PyTorchfor a machine with PyTorch

With the Hugging Face library the whole pipeline is a few calls. The processor resizes the photo, builds the chat prompt, and expands <image> into one placeholder per visual token. The model reads the configured encoder layer, drops the class token, projects, and splices.

At real scale, and the simplest design of all

LLaVA-1.5's recipe became the template for most open models. Its 13B model reached 85.9 on POPE, a test of whether a model claims objects that are not in the image (Li et al. 2023). The score is the mean of its F1 scores on the random, popular, and adversarial splits, 87.1, 86.2, and 84.5. The authors also found that raising the input resolution to 448 pixels, by encoding the image in tiles, sharply reduced hallucinated details in long descriptions. §31.6 picks up that thread.

Later models kept the encoder, projector, and language model, but changed the token count. Qwen2-VL (Wang et al. 2024a) reads images at their native resolution. Its MLP merges each $2\times2$ block of patch tokens into one, so a $224\times224$ image becomes $256/4 = 64$ tokens, plus a start and an end token, 66 in all.

At the other extreme, Fuyu-8B (Adept 2023) has no image encoder at all. Its decoder reads $30\times30$ -pixel patches. Each is flattened to $30\cdot30\cdot3 = 2{,}700$ numbers and linearly projected straight into the first layer. The patches come in raster order, left to right and top to bottom, with a special token marking the end of each row. That projection, about 11 million parameters, is the whole vision system, and it lets the model take images of any size. Everything an encoder would have learned must then be learned by the language model, which is the design space of §31.6.

📜 From a frozen prefix to visual instruction tuning

Frozen (Tsimpoukelli et al. 2021) was an early system that made a frozen language model see. It trained an NF-ResNet-50 image encoder through a frozen 7-billion-parameter model, mapping the pooled image vector linearly to $n$ prefix embeddings. The authors tried 1, 2, and 4 and found 2 worked best, and the system could learn new words from a few interleaved image-text examples.

ClipCap (Mokady et al. 2021) mapped one CLIP vector to a short prefix of vectors for GPT-2, 10 of them with its MLP mapping network. With a small Transformer as the mapping network, it captioned well even with CLIP and GPT-2 both frozen.

MiniGPT-4 (Zhu et al. 2023) put a single linear layer between a frozen Vicuna and BLIP-2's frozen visual encoder, a ViT followed by a Q-Former that turns each image into 32 vectors (§31.3). It trained that layer on about 5 million pairs in about 10 hours on 4 A100s, then fine-tuned it on 3,500 curated detailed descriptions in 7 minutes on one A100.

LLaVA, released the same month, passed all 256 patch tokens instead of one pooled vector or 32 query outputs. It unfroze the language model in a second stage and made the instruction data its main contribution.

Where it breaks

Every image costs hundreds of tokens in every layer. Eight photos at 576 tokens each are 4,608 tokens, longer than Vicuna-v1.5's 4,096-token context, so LLaVA-1.5 cannot read them at all. With an 8K-context model they fit. But in a Vicuna-7B-sized model they take 2.25 GiB of fp16 cache and about 60 TFLOPs of prefill matmuls. Few-shot prompting puts worked examples in the prompt before the real question (§6.8). With 32 example photos it would need 18,432 visual tokens, out of reach.

Two remedies follow. One keeps the image out of the sequence and lets the language model look at it through new cross-attention layers (§31.2). The other compresses each image into a few dozen tokens first (§31.3).

A 336-pixel encoder cannot read small text. A receipt, a chart, or a screenshot resized to $336\times336$ loses the detail that matters. More pixels mean more patches. The remedies, tiling, native resolution, and merging, are the subject of §31.5.

The projector cannot restore what the encoder discarded. This is a theorem, not an observation. It uses mutual information, $I(A;B)$ , the number of bits that knowing $B$ tells you about $A$ (§0.5).

∴ A connector cannot add information

Let the image be a random variable $x_{\text{img}}$ drawn from the data distribution. In this box the image is $x_{\text{img}}$ rather than the chapter's $I$ , so it never meets the mutual information $I(\cdot\,;\cdot)$ . Let $Y$ be any property of the image, such as the answer to a question about it. For any deterministic connector $g$ , the visual tokens carry no more information about $Y$ than the encoder's features do,

I\big(Y;\, g(f_{\text{v}}(x_{\text{img}}))\big) \le I\big(Y;\, f_{\text{v}}(x_{\text{img}})\big).

In particular, if two images have identical features, $f_{\text{v}}(x_{\text{img}}) = f_{\text{v}}(x_{\text{img}}')$ , then the model's answer distributions for them are identical for every prompt.

Proof. The visual tokens are computed from the features alone, so $Y \to f_{\text{v}}(x_{\text{img}}) \to g(f_{\text{v}}(x_{\text{img}}))$ is a Markov chain. The data processing inequality (§0.5) says that anything computed from the middle of such a chain carries no more information about the start than the middle itself, which is the bound. For the second claim, the language model's input depends on the image only through $g(f_{\text{v}}(x_{\text{img}}))$ , so equal features give equal inputs and equal outputs. ∎

One study turned this theorem into a benchmark (Tong et al. 2024a). The authors collected CLIP-blind pairs, two images that CLIP ViT-L/14 embeds with cosine above 0.95 but that DINOv2 tells apart. For each pair they wrote a question about the visible difference (§30.6 describes the construction). The benchmark, MMVP, counts a pair as right only if both of its questions are answered correctly, so guessing scores 25 percent. Humans reached 95.7 percent, GPT-4V 38.7, and LLaVA-1.5 24.7, below guessing. No projector can fix that, because the two images arrive almost as one.

The fixes change the features. Tong et al. interleaved DINOv2 tokens with CLIP tokens, 256 of each at 224 pixels. That raised LLaVA-1.5's MMVP score from 24.7, with 576 CLIP tokens at 336 pixels, to 28.0, with fewer tokens and a lower resolution. Prismatic concatenated DINOv2 and SigLIP features channel by channel. That keeps the number of tokens, widens only the projector's input (1,024 plus 1,152 numbers per patch), and gained 5 to 10 percent on localization and challenge sets. Cambrian-1 (Tong et al. 2024b) compared more than 20 vision encoders inside the same LLaVA-style pipeline.

Prismatic also found that fine-tuning the vision encoder during VLM training hurt in its setting, especially on localization. That is one reason LLaVA-style recipes keep the encoder frozen. Later recipes often train it gently instead (§31.4).

The table puts the connector designs of this chapter side by side.

connector	tokens per 336-px image	new parameters (7B LM)	stage 1 trains	strength	weakness	used by
linear projector	576	4.2M	the projector	simplest, keeps every patch	no capacity for nonlinear alignment	LLaVA (MiniGPT-4 used one after a Q-Former)
MLP projector	576	21M	the projector	small gain for little cost, the default	cost grows with patches	LLaVA-1.5, most open VLMs
merge then MLP	144 for a 2 by 2 merge	34M (input width $4d_{\text{vis}}$ )	the merger and projector	four times fewer tokens	neighbors share one token	Qwen2-VL, InternVL (§31.5)
learned queries	32 to 256, fixed	about 190M	the compressor	fixed small budget per image	a bottleneck for fine detail	BLIP-2 (32), Qwen-VL (256) (§31.3)
cross-attention layers	none in the sequence	about 1.6B for 8 blocks (one after every fourth of a 32-block LM, as in Llama 3.2 11B Vision, §31.2)	the new layers	language model can stay frozen	many new parameters, needs interleaved data	Flamingo, Llama 3 vision (§31.2)
no encoder	one per raw patch	about 11M	everything	any resolution, one network	must learn vision from scratch	Fuyu-8B, early fusion (§31.6)

The parameter counts are for a 4,096-wide language model. The first three rows follow from the projector formulas. The learned-query row is BLIP-2's Q-Former, and Flamingo's resampler is similar. The cross-attention row uses the $12d^2$ per block of §4.6, and the last row is Fuyu-8B's patch projection.

⚠️ Bugs that train fine and see nothing

A misaligned splice is the classic silent failure. Suppose features go to the wrong row of the batch, because images were numbered per row instead of in reading order across the batch. The loss still falls, because the language model learns to ignore the image and caption from the text alone. The model then describes pictures it was never shown.

Other bugs of the same kind follow.

Leaving image slots as targets trains the model to predict the placeholder id hundreds of times per image. That teaches nothing, and those easy targets swamp the averaged loss of the real answer tokens.
Forgetting to shift the labels by $n_v - 1$ after expansion trains each token to predict the wrong neighbor.
In a 1D-RoPE model like LLaVA, positions that restart at each image, or that count pads under left padding, give every later token a wrong rotary angle. A multimodal-RoPE model has its own rule, and advancing it by the token count is the same kind of bug (§31.5).
Silent truncation to the model's maximum length can cut off the answer. If it cuts through an image's placeholders, the features no longer match the slots and every later image shifts, so raise an error on any mismatch.
Detaching the visual features, or computing them in a no_grad block that also wraps the projector, leaves the projector at its random initialization while the language model learns to ignore it.

A quick test catches most of these. Run the model on a batch with the images shuffled among the rows. The loss should rise. If it does not, the model is not using the images.

🧭 What to remember

A late-fusion VLM encodes the image into patch features $\mathbf{Z} = f_{\text{v}}(I) \in \R^{N_p\times d_{\text{vis}}}$ , maps them with a connector to visual tokens $\mathbf{X}_v = g(\mathbf{Z}) \in \R^{n_v\times d}$ , and splices them into the language model's input, where they are read like words but never predicted.
LLaVA's connector is a linear map. LLaVA-1.5's is a two-layer GELU MLP with $d_{\text{vis}}d + d + d^2 + d$ parameters, about 21 million for a 7B model. Both read CLIP's second-to-last layer and drop the class token.
The splice must move labels, attention mask, and positions with the features. Image slots and user tokens carry the label $-100$ , and features are assigned in reading order across the batch. In a 1D-RoPE model like LLaVA, positions run through the image slots (multimodal RoPE has its own rule, §31.5).
Stage 1 trains only the projector, whose gradient through the frozen language model is $\mathbf{Z}^\top\mathbf{G}_v$ . It needs the language model's full backward pass for activation gradients, but no weight gradients or optimizer state, so it costs about $4N$ FLOPs per token instead of $6N$ .
One 336-pixel image is 576 tokens. For a 7B model that is about 7.5 TFLOPs of prefill matmuls and 288 MiB of fp16 cache, and for 13B about 15 TFLOPs and 450 MiB, read again at every decoding step.
A connector cannot add information the encoder discarded (the data processing inequality). That is why CLIP-blind pairs defeat LLaVA-1.5, and why better, fused, or higher-resolution encoders matter more than connector design.
Every image puts hundreds of tokens through every layer, so eight photos or a few-shot prompt overflow the context and the cache. §31.2 keeps the image out of the sequence altogether and lets a frozen language model look at it through gated cross-attention.

Liu et al. (2023a). Visual Instruction Tuning. LLaVA. Section 4 gives the architecture and the two training stages, Section 3 the GPT-4-generated instruction data, and Table 8 the choice of encoder layer.
Liu et al. (2023b). Improved Baselines with Visual Instruction Tuning. LLaVA-1.5. Table 2 adds one change at a time (VQA data, format prompts, the MLP, resolution), the cleanest ablation of the recipe.
Tsimpoukelli et al. (2021). Multimodal Few-Shot Learning with Frozen Language Models. Frozen, an early model that trained a vision encoder through a frozen language model and treated images as a prefix of embeddings.
Mokady et al. (2021). ClipCap: CLIP Prefix for Image Captioning. A mapping network from one CLIP vector to a GPT-2 prefix, with both large models frozen, a direct ancestor of the projector.
Zhu et al. (2023). MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. One linear layer between a frozen visual encoder and a frozen Vicuna, and why a second stage on detailed descriptions was needed.
Karamcheti et al. (2024). Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. Controlled experiments on single-stage training, freezing the encoder, image preprocessing, and fusing DINOv2 with SigLIP.
Tong et al. (2024a). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CLIP-blind pairs, the MMVP benchmark, and how an encoder's blind spots become a VLM's.
Tong et al. (2024b). Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. More than 20 vision encoders compared inside one LLaVA-style pipeline, with a vision-centric benchmark.

Made for larger screens

Connecting a vision encoder to an LLM

A language model reads vectors, not words

One photo, end to end

The late-fusion model, defined

Which features, and how many

What one image costs

What an image costs

The projector

The LLaVA-1.5 projector

Splicing image features into the sequence

Splice image features into a batch

Training LLaVA in two stages

Train a projector through a frozen head

How simple can the connector be

A closed-form connector

In PyTorch

At real scale, and the simplest design of all

Where it breaks

Read more

Watch next