This section is about a second way to let a language model see. In the first way, image vectors join the model's input sequence and every layer processes them like words. In this one, the sequence stays text, and new layers inserted into the model let the text read the image. The design comes from Flamingo (Alayrac et al. 2022), and it matters for two reasons. It can keep a pretrained language model exactly as it was, so its text skills cannot be damaged. And an image costs work in a few new layers instead of hundreds of extra positions in every layer.

Recall the first way, from §31.1. A frozen image encoder turns a photo into one vector per patch of the image. A small network, the projector, maps each vector to the language model's width, and the results are spliced into the input sequence as if they were word embeddings. We will call this the decoder-only design, because one decoder-only language model (§6.2) reads the image tokens and the text together. LLaVA-1.5 (Liu et al. 2023b) is the standard example, with $n_v = 576$ visual tokens per image.

Now give that model a web page with five photos. The sequence grows by $5\times 576 = 2{,}880$ positions before a single word is read. Every one of them passes through every layer's projections and feed-forward network, at about $2N$ floating-point operations (FLOPs) per token for a language model with $N$ weights. Every one also stores a key and a value in every layer's KV cache, the memory that keeps past keys and values during generation (§6.5). A prompt with 32 captioned photos as examples carries 18,432 visual tokens. And to follow visual instructions well, §31.1 fine-tuned the language model itself, which puts its text skills at risk.

So here is the question of this section. Can we keep a strong language model frozen, with its weights untouched, and still let it see, at a cost that does not grow with every image in every layer?

Flamingo answered yes with three pieces, and this section builds each one.

• Gated cross-attention layers. New layers sit between the frozen model's layers, and in them the text reads the image tokens, which never join the sequence. Each new layer's output is multiplied by a learned gate that starts at zero, so at the start of training the model is exactly the original language model.
• A Perceiver resampler. A small network that squeezes any image, or any video, into a fixed set of $m = 64$ visual tokens.
• An image-causal mask. A rule that tells each text token which image to read. It lets one model read web pages in which images and text alternate, and learn a new task from a few examples in its prompt.

Along the way we prove what the zero gate does to the forward pass and to the gradients, count what cross-attention costs against putting images in the sequence, and read the controlled studies that decided which design the field kept.

§31.1 sorts every design in this chapter by three questions. Where does visual information enter the model? How many visual vectors per image does the model handle? What is trained, and what stays frozen? Flamingo's answers are these. Vision enters through new cross-attention layers inside the language model. Each image becomes $m = 64$ vectors, and only the new layers read them. Only the resampler and the new layers are trained, while the vision encoder and the language model stay frozen.

Cross-attention with one text token and two visual tokens

Cross-attention is attention between two sequences (§4.6). One sequence asks the questions, so each of its positions has a query vector. The other sequence, called the context, holds the answers. Each context position has a key, which is compared with the query, and a value, which is what gets averaged. The output has one row per query, the average of the values weighted by how well each key matches.

Let the queries come from the text and the context be the image's visual tokens. Then each text position gets back a weighted average of what the image says. Add that average to the text's residual stream, the running vector at each position that every layer reads and adds its own result to (§5.1). From then on the text carries visual information into every later layer.

The second idea is the gate. Multiply the new branch's output by a learned number before adding it, and start that number at zero. Then the new layer does nothing at first. The frozen model is untouched until training decides, one step at a time, how much of the image to let in. Flamingo passes the number through $\tanh$ first, for reasons we come to below.

In practice the zero gate means you can attach vision to a language model you already trust and know that, before any training, its text behavior is unchanged. Why the gate starts at zero, and why it goes through $\tanh$, are two of the first things an interviewer asks about this design.

The gated cross-attention block, defined

Flamingo inserts one new block in front of the first frozen block of the language model (the LM from here on) and then in front of every $k_x$-th one. The spacing $k_x$ is a design choice. Flamingo's three models use 1, 4, and 7. Each new block has two branches, and each branch has its own gate. A cross-attention branch reads the image. A feed-forward branch, a two-layer network applied to each position separately, processes what was read. The paper calls the block GATED XATTN-DENSE.

In plain words, each new block lets the text read the image, then process what it read, and two gates decide how much of each result is added to the stream the frozen LM sees. Both gates start at zero.

Notice what the block does not do. The visual tokens never enter the sequence, so the frozen self-attention and feed-forward layers see $T$ positions, exactly as they would for text alone. And the LM never updates the visual tokens. They are read, layer after layer, but nothing writes to them.

The cell implements one block with a single head. It checks that the block is the identity at zero gates, on a sequence whose first row sees no image.

The identity is exact, bit for bit. Now look at row 0 of the second line. It sees no image, yet it moved by 0.143. The reason is that the gated feed-forward branch acts on every position, whether or not it read an image. So in Flamingo's formulation, once the gates open, the new layers change the LM's computation even on text that comes before every image. Meta's reference code for Llama 3.2 Vision avoids this. On rows that see no image it multiplies the output of both branches by zero, so such text is processed exactly as by the frozen LM. For a prompt with no image at all, it skips the new layers.

import numpy as np

def gated_xattn_dense(Y, Xv, mask, params):
    """params holds W_Q (d, d_k), W_K (d_c, d_k), W_V (d_c, d_k), W_O (d_k, d),
    W_1 (d, d_ff), W_2 (d_ff, d), and the scalars alpha_xattn and alpha_dense."""
    raise NotImplementedError

Why the gates start at zero, and what that does to the gradients

The zero gate makes two promises. One is about the forward pass, and one is about the backward pass. Both are exact, and the second is the one interviewers probe. Two pieces of notation make them short. Write $\langle\mathbf{A}, \mathbf{B}\rangle = \sum_{ij}A_{ij}B_{ij}$ for the sum of the entrywise products of two matrices of the same shape. And write $\mathbf{U}_x = \operatorname{xattn}(\mathbf{Y}, \mathbf{X}_v; \mathbf{M})$ and $\mathbf{U}_d = \operatorname{ffw}(\mathbf{Y}_1)$ for the outputs of the two branches.

In plain words, at initialization the gates are the only new parameters with a gradient. A gate's gradient, $\langle\mathbf{G}, \mathbf{U}\rangle$, says how much the loss would change if a little of its branch's current output were let into the stream. Every weight behind a gate waits until that gate has moved away from zero.

The cell checks the theorem by finite differences on the block above. A finite difference estimates a derivative by nudging one parameter up and down by a tiny amount and measuring the change in the loss.

The finite difference for the gate matches $\langle\mathbf{G}, \mathbf{U}_x\rangle$ to six digits. The inner weights get exactly zero. And as soon as the gate is 0.3, the same weight gets a gradient.

In practice the theorem tells you what to check on the first step of a run. The gates must have nonzero gradients and every new weight must have exactly zero. If an inner weight has a gradient at step 0, a gate is not where you think it is.

The first steps, under gradient descent and under Adam

The theorem describes one point, the initialization. What happens in the steps after it depends on the optimizer. The two that matter are plain gradient descent and Adam. Adam keeps a running average of each parameter's gradient and of its square, and divides the first by the square root of the second, so each parameter's step is about the learning rate $\eta$ whatever the size of its gradient (§0.6). Flamingo and the other recipes in this chapter use AdamW, which is Adam with weight decay applied directly to the weights.

Plain gradient descent first. Step 1 sets $\alpha_{\text{xattn}} = -\eta\langle\mathbf{G}, \mathbf{U}_x\rangle$, which is generally nonzero, and leaves every new weight where it was. On step 2 a new weight $w$ gets the gradient $\tanh(\alpha_{\text{xattn}})\langle\mathbf{G}_1, \partial\mathbf{U}_x/\partial w\rangle$. That is of order $\eta$, because $\tanh\alpha_{\text{xattn}} \approx \alpha_{\text{xattn}}$ is of order $\eta$. So the weight moves by an amount of order $\eta^2$, and the inner weights start slowly while the gates open.

Adam agrees on step 1 and disagrees from step 2. On step 1 a weight whose gradient is exactly zero gets zero averages and a zero update. (AdamW's weight decay still multiplies it by $1 - \eta\lambda$, with $\lambda$ the decay rate.) The gate's first step is a sign step of size $\eta$, so $\alpha_{\text{xattn}} = -\eta\operatorname{sign}\langle\mathbf{G}, \mathbf{U}_x\rangle$. On step 2 the weight's gradient is small, of order $\eta$. But Adam divides each gradient by its own root mean square, and the smallness cancels.

In plain words, the size of $g_2$ cancels because Adam divides each gradient by its own running root mean square, the normalization of §0.6. So a weight that waited one step then moves almost a full learning rate at once. Two details of real optimizers could change that conclusion, the $\epsilon$ in the denominator and the way the step counter advances, and the box shows that neither does.

So under Adam the inner weights move at the full scale of the learning rate from step 2 on, about $0.74\eta$ per coordinate, the same order as the gate. The zero gate does not make them learn slowly. It still guarantees three things.

• The forward pass at initialization is exactly the frozen LM.
• The inner weights wait one step, so their first move follows a gradient computed with the gate already turned in a useful direction.
• Whatever the inner weights do reaches the output only through $\tanh\alpha$, which stays small for many steps.

The last point deserves numbers. Adam moves the gate by at most about $\eta$ per step, so after $k$ steps $\lvert\tanh\alpha\rvert \le \lvert\alpha\rvert$ is at most about the sum of the first $k$ learning rates. Flamingo's rate rises linearly from 0 to $10^{-4}$ over its first 5,000 steps. So after 100 steps the gate is at most about $10^{-4}$, and by the end of the warmup at most about $5{,}000\times 10^{-4}/2 = 0.25$. Near zero, the change in the branch's contribution is $\Delta(\tanh(\alpha)\,\mathbf{U}_x) \approx \Delta\alpha\,\mathbf{U}_x + \alpha\,\Delta\mathbf{U}_x$, and the second term is a product of two small factors. The weights behind the gate move at full speed, but the image enters the stream gradually, through the gate.

The cell runs both optimizers on the small example of Exercise 31.2e below, four text positions reading three visual tokens through a trainable value matrix.

With $\eta = 10^{-3}$, gradient descent moves $\mathbf{W}_V$ by at most $0.0008\eta$ on step 2, a move of order $\eta^2$. Adam moves it by exactly $0.7441\eta$ in every coordinate, the value of the theorem, and moves the gate by $\eta$ per step. An interviewer who asks "which parameters learn first" often follows with "and does that change under Adam?", and this is the answer.

The same trick as LoRA, and a trap to avoid

The zero gate has the same structure as the initialization of LoRA (§14.5). LoRA fine-tunes a frozen weight matrix by adding a low-rank update $\gamma\mathbf{B}\mathbf{A}$, the product of two thin matrices scaled by a constant $\gamma$, and it starts with $\mathbf{B} = 0$. So the model starts as the base model, $\mathbf{A}$'s gradient is zero (it is proportional to $\mathbf{B}$), and only $\mathbf{B}$ moves on the first step. The tanh gate plays the role of $\mathbf{B}$, one number per branch instead of a matrix, and the branch's weights play the role of $\mathbf{A}$.

The same warning applies. Suppose the branch output were also zero at initialization, for instance because someone also zero-initialized $\mathbf{W}_O$. Then $\langle\mathbf{G}, \mathbf{U}_x\rangle = 0$ and the gate's gradient vanishes. Every inner gradient is still multiplied by $\tanh 0$, so nothing ever moves. The starting point is a saddle, a point where every gradient is zero but which is neither a minimum nor a maximum. Near it the loss changes by a term proportional to the product of the gate and $\mathbf{W}_O$, which goes up in some directions and down in others. LoRA avoids the same saddle by initializing $\mathbf{A}$ randomly. Exactly one factor of the product may start at zero.

Why tanh rather than a raw scalar

A raw scalar gate, $\alpha\,\mathbf{U}$ with $\alpha = 0$ at the start, would keep both promises of the theorem too. The two gates agree to first order near zero, since $\tanh'(0) = 1$. What tanh adds is a bound and a brake.

• A bound. Because $\lvert\tanh\alpha\rvert < 1$, the gate can never amplify its branch, only pass it through. Any growth in the visual contribution must come from the branch's own weights, where weight decay and the optimizer's normalization act on it.
• A brake. The gate's gradient carries the factor $1 - \tanh^2\alpha$, which shrinks as the gate opens. Under gradient descent that damps the gate's own steps. Under Adam the normalization divides most of it away.

That is a design argument, not a theorem. The evidence is an ablation that removes the zero-initialized tanh gate altogether, adding the new branches ungated (Flamingo's Table 3, row iii). The overall score falls from 70.7 to 66.5, and the authors report training instabilities without the gate. The paper does not compare tanh with a raw scalar. Its Figure 6 shows the gates of Flamingo-3B leaving zero quickly in every layer during training, with larger absolute values deeper in the network. The authors decline to interpret that trend, because the scale of the branch outputs also varies with depth.

The widget trains the small gated branch of Exercise 31.2e by gradient descent, with $\alpha = 0$ at the start. The left panel, the forward pass, shows the gate value $g$ and the change from frozen, $\lVert\mathbf{Y}' - \mathbf{Y}\rVert_F$, how far the output has moved from the frozen model's. A dashed line marks the change the target asks for. The right panel, the backward pass, shows the sizes of the two gradients, the gate's and the value matrix's.

import numpy as np

def gate_grads(alpha, W_V, Y, A, Xv, Y_star):
    raise NotImplementedError

def train_gate(alpha, W_V, Y, A, Xv, Y_star, lr, steps):
    raise NotImplementedError

The Perceiver resampler

The gated layers read a set of visual tokens, but which set? A frozen vision encoder produces one feature vector per patch, and the number of patches grows with the image's resolution. A video multiplies it by the number of frames. Flamingo wants every image and every clip to cost the gated layers the same, so it compresses each one to a fixed $m = 64$ tokens with a Perceiver resampler.

The idea comes from the Perceiver (Jaegle et al. 2021), which §4.6 introduced. A small array of learned vectors, called latents, serves as the queries of a cross-attention whose context is the large input. The cost grows only linearly with the input's size, and the output has one row per latent, whatever the input's length. With a single latent this is attention pooling with a learned query, built in Exercise 4.6c, and a full Perceiver encoder is built in Exercise 4.6f. Flamingo's version differs in two details, both visible in the definition.

In plain words, 64 learned questions are asked of the image six times, each time refined by the answers so far, and the 64 final answers are the image's tokens.

The first difference from a plain Perceiver is in the keys and values. They come from the features and the latents together, so in one softmax a latent attends both to the image and to the other latents. The same attention does double duty as the latents' self-attention. The paper's appendix reports that this did slightly better than keys and values from the features alone, as in the Perceiver and in the object detector DETR (Carion et al. 2020).

The second difference is that only temporal embeddings are added, and no spatial position embeddings. The authors saw no gain from explicit spatial embeddings. Their suggested reason is that the features of a convolutional network, such as their image encoder NFNet, already carry spatial position in their channels (Islam et al. 2021).

Note also that $\mathbf{X}_f$ is never updated. A Transformer run over the features would rewrite them layer after layer. Here every layer re-reads the same features, just as the Perceiver's repeated cross-attentions do.

Why does the resampler need temporal embeddings at all? Because cross-attention cannot tell the order of its context, and the next result makes that precise.

In plain words, to the resampler a clip is a bag of patch features, each tagged with its frame number. It cannot see where in the frame a patch was, unless the features themselves say so. It knows when a patch was only through the tag.

Shuffling patches changes the output only by rounding error, about $10^{-15}$, while playing the video backwards changes it by more than 1. The invariance has a cost that matters for documents. Each output token is a weighted average over all patches, so the resampler mixes patches from all over the image. Fine spatial layout survives only as far as the encoder wrote it into the features.

NVLM (Dai et al. 2024) found exactly this. A Perceiver resampler helped captioning of natural images but hurt dense OCR (reading the text in scanned documents), so NVLM's cross-attention model, NVLM-X, drops the resampler and cross-attends directly to the encoder's tokens. Qwen-VL (Bai et al. 2023) keeps a one-layer resampler, 256 learned queries that compress the features of a Vision Transformer (ViT) before a decoder-only LM. It adds 2D absolute position encodings to its query-key pairs so that positions survive the compression. §31.3 asks what such learned queries should be trained to keep.

Temporal embeddings also handle videos longer than those seen in training. Flamingo trained on clips of 8 frames sampled at one frame per second. At inference it fed 30 frames at 3 frames per second, by linearly interpolating the 8 learned temporal embeddings to 30 (Appendix B.1.2 of the paper). ViTs use the same trick for position embeddings at a new resolution (§27.1). Count frames from 0 here, so the old embeddings are numbered 0 to $F - 1$. With the endpoints aligned, new frame $k$ of $F'$ sits at old position $k(F - 1)/(F' - 1)$ and takes the linear blend of its two neighbors. For $F = 8$ and $F' = 30$, new frame 1 sits at old position $7/29 = 0.241$, so it is 76 percent of embedding 0 and 24 percent of embedding 1.

What do the ablations say about the resampler itself? At an equal parameter budget, replacing it by an MLP or by a plain Transformer over the features lowered Flamingo-3B's overall score from 70.7 to 66.6 and 66.7, and both alternatives were slower (Table 3, row vi). A smaller resampler scored 67.9 and a larger one 69.0. Scaling the resampler up together with the LM made training unstable, so all three Flamingo models share the medium one (Table 10).

import numpy as np

def perceiver_resampler(X_f, E_t, X_q, layers, h):
    raise NotImplementedError

Interleaved documents and the image-causal mask

A caption dataset has one image and one sentence per example. A web page is different. It is an interleaved document, paragraphs and images in a reading order, and a sentence may describe the image above it, the image below it, or neither. Flamingo models such a document as text with images at known positions. It predicts each text token from the text before it and from the images before it,

Here $y_\ell$ is the text token at position $\ell$, $y_{\lt\ell}$ are the tokens before it, $I_1, \dots, I_{N_{\text{img}}}$ are the document's images in reading order, and $I_{\le\ell}$ is the set of images whose <image> tags lie at or before position $\ell$.

In plain words, the model writes the page one token at a time and may look at any picture it has already passed.

The mask defined below narrows this further inside the gated layers. Position $\ell$'s cross-attention reads only the last of these images. The earlier ones reach it only indirectly, through the frozen self-attention over earlier text positions that did read them.

To build the training data, Flamingo scraped about 43 million web pages into a dataset called M3W. It inserted an <image> tag in the text wherever an image appeared, and a learned <EOC> ("end of chunk") token before each image and at the end of the document. Each training example is a random 256-token window of a page with up to its first 5 images. Image-text pairs are written in the same format, <image> caption <EOC>.

Which image should a text token's cross-attention read? Flamingo's answer is the most recent one.

In plain words, every text token looks at the last picture it has passed, and only that one.

The tag position itself sees its image. That is a choice, so it needs a reason. In a decoder the hidden state at position $\ell$ predicts the token at $\ell + 1$, so the tag's hidden state predicts the first word after the image, the word most in need of seeing it. Flamingo's Figure 7 marks the tag with the new image's index, and OpenFlamingo's code implements this "at or before" rule with a running count over tag positions.

Why only one image, when the earlier ones may matter too? There are three reasons, and the last one was measured.

• It generalizes across the number of images. Whatever the document, each row sees exactly $n_v$ visual tokens, so a model trained on at most 5 images per sequence meets nothing new when a prompt has 32. Under the cumulative mask, a row late in a 32-image prompt would see more than six times as many visual tokens as any row seen in training. Flamingo trained with at most 5 images and was evaluated with up to 32.
• Earlier images are still reachable. The text tokens after image 1 have read it, and the frozen self-attention lets every later token read those text tokens. So information from image 1 reaches the text about image 2, one step removed, without image 1's visual tokens being attended directly.
• It worked better. Attending to all previous images lowered Flamingo-3B's overall score from 70.7 to 63.5 (Table 10, row ii). The authors suggest that the cross-attention cannot tell the images apart when they are pooled. They report that marking the images with indexed tags or index embeddings was less robust when the number of images changed between training and test.

There is one more twist. On a web page, text often describes the image that follows it ("Here is my dog," then the photo). So for half of the training pages, chosen at random, Flamingo mapped each text token to the next image instead of the previous one ($p_{\text{next}} = \frac12$). That scored 70.7, a small win over always previous (69.6) or always next (70.4). Attending to a later image is not a leak, because images are inputs only and never prediction targets.

In the widget, each row is a position of the sequence and each column a visual token, grouped and colored by image. The column $\varphi$ counts the image tags at or before each row. Click a word to insert an image before it, or click an <image> row to remove it. The "next image" mode maps each row to the image after it, the rule Flamingo used on half its pages. The widget starts with two images of $m = 4$ tokens. Before you click, predict the most visual tokens any row can see in "last image" mode. Then switch to "all previous images" and watch what that maximum does as you insert a third image and then a fourth.

Look also at the rows that see nothing in either mode, the text before the first image. Those empty rows are where implementations of this mask most often break.

import numpy as np

def image_causal_mask(tokens, m, image_token="<image>"):
    raise NotImplementedError

Few-shot prompting a vision-language model

Interleaving is what turns a captioner into an in-context learner. A language model given a few input-output examples in its prompt continues the pattern for a new input, with no change to its weights. That is in-context learning (§6.8), and Flamingo does the same with images. A captioning prompt is

<BOS><image>Output: A cat wearing sunglasses.<EOC><image>Output: Elephants walking in the savanna.<EOC><image>Output:

with the three images fed to the resampler. The model's continuation, up to its first <EOC>, is the caption of the third image. Visual question answering uses Question: ... Answer: ... in the same way.

Here is how the mask makes this work. Each example's text reads its own image, and the query image is read by the tokens after its tag. The frozen self-attention lets the query text read the earlier examples' text, which is where the pattern lives.

For its "zero-shot" numbers Flamingo gave two text-only examples, with their images removed. The authors' reasoning is that a hand-written task description would need labeled examples to validate, so it would not be truly zero-shot. They also found that one text example made the model copy its style, while more than two helped only marginally.

A model trained only on single image-caption pairs never sees a sequence like this one. M3W is what provides it. Removing M3W was the largest single loss in Flamingo's ablations, from 70.7 to 53.4 overall (Table 3, row i), against 60.9 for removing the image-text pairs. Interleaved data has since been rebuilt in the open as OBELICS, 141 million web pages from Common Crawl with 353 million images and 115 billion text tokens. It trained the open IDEFICS models at 9B and 80B parameters, which follow Flamingo's design (Laurençon et al. 2023). OpenFlamingo's 3B to 9B models reach 80 to 89 percent of the corresponding Flamingo scores on seven benchmarks (Awadalla et al. 2023). §31.4 treats interleaved documents as one ingredient of a training mixture.

What cross-attention costs, against putting images in the sequence

Now we count both designs exactly. §4.6 compared them with a simplified model (full multi-head attention and an MLP of width $4d$). Here is the exact count for a modern LM, whose setup has four parts.

• Attention. The LM uses grouped-query attention, in which groups of query heads share one key head and one value head, so it has $h$ query heads but only $n_{kv}$ key-value heads, all of width $d_k$ (§12.5).
• Feed-forward network. It is a SwiGLU network, a gated feed-forward network with three weight matrices, $3dd_{\text{ff}}$ weights for hidden width $d_{\text{ff}}$.
• Counting convention. As in §5.6, we count 2 FLOPs per multiply-add and only matrix products. Prefill is the forward pass over the whole prompt before the first output token.
• Shapes. The LM has $L$ blocks of width $d$. Each block has attention weights $P_{\text{att}} = 2dhd_k + 2dn_{kv}d_k$ (query and output projections, then keys and values) and feed-forward weights $P_{\text{ffn}}$. One query-key pair costs $2hd_k$ FLOPs for its score and $2hd_k$ for its share of the weighted sum.

Let $n_v$ visual tokens, already at width $d$, accompany $n_t$ text tokens. Let the gated blocks have the shapes of an LM block, one after every $k_x$ LM blocks, with $k_x$ dividing $L$ so that there are $L/k_x$ of them.

In plain words, putting an image in the sequence costs a whole forward pass per visual token. Cross-attention costs a key and a value per visual token in a few layers, plus extra work on every text token in those layers. So cross-attention wins on compute only when the image is long compared with the text, roughly when $n_v$ exceeds $n_t/k_x$.

The cell evaluates the theorem for the text model of Llama 3.2 11B Vision. Its released configuration has exactly this shape, with $d = 4{,}096$, 32 self-attention blocks with $h = 32$, $n_{kv} = 8$, $d_k = 128$, and $d_{\text{ff}} = 14{,}336$, and 8 gated cross-attention layers among them, one for every 4 self-attention blocks. We take a 200-token prompt and four image sizes. They run from a 64-token resampled image to the 6,404 tokens that model's encoder produces for an image of four 560-pixel tiles ($4\times(40^2 + 1)$, counting one summary token per tile).

Generation tells a different story. After prefill, each new token runs the 8 gated blocks too. At batch size 1, generating a token is limited by how many bytes must be read from memory, mostly the weights (§20.1). The cross-attention model reads 1.745B more weights per step but far less cache, since the visual keys and values live in 8 layers instead of 32. Which effect wins depends on the batch, as the cell shows for a 2,880-token image.

At batch 1 the cross-attention model reads about 20 percent more bytes per token, so it is slower. Past a batch of about 12 requests, each with its own image, the cache dominates and cross-attention is faster, reading 26 GB per step against 41 GB at batch 64. An interviewer who asks for the saving expects both answers, prefill cheaper by the image-to-text ratio and generation slower at small batch.

def prefill_flops_decoder_only(cfg, n_v, n_t):
    raise NotImplementedError

def prefill_flops_cross(cfg, n_v, n_t, every):
    raise NotImplementedError

def kv_bytes_visual(cfg, n_v, every, design, bytes_per=2):
    raise NotImplementedError

What the controlled comparisons say

Compute is one column of the ledger. Quality is the other, and it can only be read from experiments that hold everything else fixed. Three sets of them shaped the field.

Idefics2's controlled comparison. Laurençon et al. (2024a) built both designs on the same backbones with the same data. Their fully autoregressive design is the decoder-only design, here with a 64-token Perceiver resampler whose output is projected into the sequence. They scored each design by its average 4-shot score on four benchmarks, VQAv2, TextVQA, and OKVQA (question answering, scored by accuracy) and COCO captioning (scored by CIDEr).

• With frozen backbones, cross-attention won, 66.7 against 60.3. It had 1.3B more trainable parameters (2B in total) and used 10 percent more FLOPs at inference.
• With the backbones adapted by LoRA (§14.5), the fully autoregressive design rose 9.2 points to 69.5, and cross-attention only 0.6 to 67.3. (These are the numbers of the paper's Table 3. Its prose gives 12.9 for the first gain.)
• Fully unfreezing the fully autoregressive model made training diverge, and LoRA fixed that.

Their Finding 2 states the conclusion. Cross-attention is better when the backbones are frozen, and worse once they are trained, despite having more parameters.

NVLM's three architectures. Dai et al. (2024) trained three models on the same data. NVLM-D is decoder-only. NVLM-X uses cross-attention, with no resampler and a gated cross-attention layer every 8 LM layers of Qwen2-72B. NVLM-H is a hybrid.

• NVLM-X trained much faster, 50.6 samples per second against 28.8 for NVLM-D, on a 34B model with seven 256-token tiles per image, because the decoder never processes the 1,792 image tokens.
• NVLM-D read better, 92.6 against 82.9 on DocVQA (questions about scanned documents) and 853 against 828 on OCRBench, a text-reading benchmark, at 72B.
• The hybrid feeds a low-resolution thumbnail's tokens into the decoder for joint reasoning and sends the high-resolution tiles through cross-attention.
• Freezing the LM during multimodal fine-tuning cost NVLM-X accuracy (DocVQA 76.4 frozen against 82.9). So NVLM unfroze the LM and mixed in high-quality text-only data to keep its text skills.

Llama 3's vision adapter (Llama Team 2024) is the largest cross-attention vision-language model described in detail.

• Cross-attention layers sit after every fourth self-attention layer, about 100B parameters on the 405B model.
• The encoder is a ViT-H/14 of 630M parameters, extended with 8 more layers to 850M and read at several depths to give 7,680-dimensional features.
• The encoder is trained while the LM stays frozen through pretraining and fine-tuning, so text-only behavior is guaranteed unchanged.
• The frozen LM had a second benefit. At fine-tuning the team swapped in the instruction-tuned LM's weights in place of the pretrained ones, under the same adapter.
• Its images averaged 2,308 tokens against 192 text tokens, the regime where cross-attention's compute advantage is largest.
• To make training converge, gradients were accumulated over micro-batches in fp32.

For Llama 4, Meta moved to early fusion, with image and text tokens in one backbone from the start of pretraining (§31.6).

So why did the field move to the first two rows? Because the cross-attention design's two advantages, cheap compute on very long images and a guarantee about text, both have cheaper substitutes. Compressing or merging visual tokens (§31.3 and §31.5) attacks the compute directly. Mixing text-only data into multimodal training keeps text skills well enough in practice. Meanwhile the decoder-only design adds few parameters, reuses the LM's serving stack, lets every layer and every text token interact with every visual token, and wins once the backbones are trained.

That is an empirical trend, not a theorem. Cross-attention remains the natural choice where the context is long and the output short, as in speech recognition. The same block also runs in the other direction in text-to-image diffusion, where the image being generated cross-attends to the prompt.

In PyTorch

A gated block in PyTorch uses scaled_dot_product_attention with a boolean mask (True means may attend). Rows that see no image are handled by letting them attend to anything and then zeroing their output, which keeps the kernel free of NaN. The block below checks four things. The block is the identity at initialization. After one backward pass the gates have gradients and every weight has exactly zero, as the theorem says. One step later the weights have gradients. And a visual token hidden by the mask has no effect.

In a production model the gated blocks sit inside the decoder's list of layers. The visual keys and values of each gated block are computed once per image and cached separately from the text cache, and the gated blocks can be skipped for a prompt with no image. The released Llama 3.2 vision layers also normalize the queries and keys of the cross-attention before their dot product (QK-norm, §8.8), and they use the LM's SwiGLU network in the dense branch.

Read more

• Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. The source for everything in this section. Read Section 2 for the architecture, Tables 3 and 10 for the ablations, and Appendix A.1 for the resampler, the gated block, and the masking, with pseudo-code.
• Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention. The latent cross-attention idea behind the resampler, and the experiments showing its cost is linear in the input.
• Bachlechner et al. (2020). ReZero is All You Need: Fast Convergence at Large Depth. The residual multiplier initialized at zero that Flamingo cites for its gates.
• Awadalla et al. (2023). OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models, and Laurençon et al. (2023), OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents. The open reproductions of the model and of its interleaved data.
• Laurençon et al. (2024a). What matters when building vision-language models?. Section 3.2 and its Table 3 are the cleanest controlled comparison of cross-attention and fully autoregressive designs, frozen and with LoRA.
• Dai et al. (2024). NVLM: Open Frontier-Class Multimodal LLMs. Decoder-only, cross-attention, and hybrid models trained on the same data, with throughput, OCR, and the effect of freezing the LM.
• Llama Team (2024). The Llama 3 Herd of Models. Section 7 describes the cross-attention adapter at 405B scale, its frozen LM, and the engineering problems of training it.
• Bai et al. (2023). Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. A one-layer resampler with 256 queries and 2D positions in front of a decoder-only LM, the bridge between this section's design and the next.

Watch next

▶

▶

▶