Python · cell 1
This section is about the Q-Former, short for Querying Transformer. It is a small network that turns an image into a short, fixed list of vectors that a frozen language model can read. It is the connector of BLIP-2 (Li et al. 2023). Its idea is to teach the connector what text needs from an image, with training tasks of its own, before the connector ever meets the language model. Whether that beats a plain projector that hands over many more vectors is a choice every vision-language team makes, and interviewers ask about it often.
A connector is the trained piece between a frozen image encoder and a language model. Frozen means the weights are never updated. The two sections before this one built the two simplest connectors. In §31.1, a small projector maps the feature of every image patch into the language model's input space, so a 336-pixel image becomes 576 extra tokens in the sequence. In §31.2, Flamingo's resampler squeezes any image into 64 vectors, using 64 learned vectors that each read the whole image (Alayrac et al. 2022).
That resampler learns what to keep only from the language model's next-word loss. The signal has to travel back through a frozen 70-billion-parameter language model. Flamingo needed about 2 billion image-text pairs and 43 million web pages to learn from it. BLIP-2 asks whether a small module can first learn what text needs from an image on its own, and only then be attached to the language model. The answer was yes, by a wide margin.
This section builds the Q-Former piece by piece. We define it with every shape and follow one photo through it. Then we derive its three training tasks, which differ only in which positions may read which, and implement each one. We follow its two training stages to a frozen language model and count its parameters to the million. Then we prove what a few vectors cannot carry, and read the evidence on whether a clever compressor beats a plain projector.
A learned query is a vector, trained like any other weight, that reads a set of features through cross-attention. Here is cross-attention again, from §4.6. After learned linear projections, the query is compared with every feature by a dot product. The scores become weights that add up to one. The output is the weighted average of the features. The query decides the weights, so the output is the summary of the set that the query has learned to ask for.
Here is one query reading four patches. Say its scores against the patches are 2, 0, 0, and 0. The softmax weights are for the first patch and for each of the others. So the output is mostly the first patch. A different image gives different scores and a different summary, but the query vector is the same for every image.
Several learned queries give several summaries, and each is free to specialize. The object detector DETR uses 100 learned queries, and each learns to claim one object in the image (§27.6, Carion et al. 2020). Flamingo's resampler uses 64 of them to turn any image into 64 vectors.
The Q-Former has 32 learned queries. Think of them as 32 questions the module asks of every image. The questions are the same each time, and the answers depend on what is in the picture. What is new in BLIP-2 is how the answers are graded. Each training image comes with its caption, and the module is set three tasks.
All three tasks run in one network. The queries and the caption's tokens sit in one sequence and share the same self-attention layers. The only thing that changes between tasks is the attention mask, the rule that says which position may read which.
Each task's rule follows from what it must prevent. Take queries and caption tokens. For generation, the text must see both queries, because they are its only view of the image. Each text token must see only earlier text, or predicting the next word becomes copying it. And the queries must not see the text. Otherwise they could pass the caption through to the decoder and learn nothing about the image.
For contrastive learning, the queries and the text must not see each other at all. Otherwise the image's embedding would contain the very caption it is being compared with. For matching, everything sees everything, because judging a pair needs both at once.
First the image side. The largest BLIP-2 models read the image with the ViT-g/14 of EVA-CLIP (Fang et al. 2023). It is a Vision Transformer of about a billion parameters ("g" for giant) that cuts the image into -pixel patches, trained with a CLIP loss to match images with their captions. BLIP-2 removes its last layer and reads the features of the second-to-last, which the paper reports works slightly better. LLaVA made the same choice for CLIP (§31.1).
At pixels the encoder cuts the image into patches. It also adds one extra vector, the CLS feature, a learned summary slot that attends to every patch (§27.1). So it hands over feature vectors, each of width . With CLIP's smaller ViT-L/14 the features are .
Now the text side. BERT-base is a 12-layer Transformer encoder for text, trained by filling in masked words (§7.1). Its attention is bidirectional, so every token reads every other token. Its first token, [CLS], has an output that serves as a summary of the sentence. The Q-Former starts from BERT-base and adds the queries and a door to the image.
In plain words, the image enters through one door, the queries' cross-attention. The queries and the text meet only in the shared self-attention, where the mask decides who may read whom. The paper describes the module as two Transformers that share their self-attention layers. One is an image Transformer, the query path. The other is a text Transformer that serves as both an encoder and a decoder. The separate query FFN is visible in the released code, in Salesforce's LAVIS library, and it matters for the parameter count later.
Three consequences of the definition matter in practice.
The rest of the section explains why step 4 produces something the language model can use. The answer is in how the queries were trained.
Now make the masks precise. Number the positions with the queries first. Write when position may attend to position , and when it may not.
In plain words, ITC keeps the image side and the text side apart. ITG lets the text read the image side but not the reverse, and lets each word read only the words before it. ITM lets everything read everything.
All three are masks from §4.4. ITC is a block-diagonal mask, with the queries as one document and the caption as another, each read in both directions. ITG is a prefix-LM mask. It lets a prefix be read in both directions and makes everything after the prefix causal. Here the queries are the prefix. BLIP-2 took this idea from UniLM, which trained one language model with several masks (Dong et al. 2019, §8.1). ITM is no mask at all. The cell builds the three for and .
The masks are easy to write down. What they force is the interesting part, and it holds for the whole 12-layer stack, not just one layer.
Before layer 1 the query states are and the text states are embeddings of the text, so all three claims hold. Suppose they hold before some layer.
In the self-attention, row 's output is . Here wherever , and the nonzero weights are a softmax over the allowed scores only. For a query row the allowed are queries, so its output depends only on query states (claim 1). For a text row under ITC the allowed are text, so its output depends only on text states (claim 2). For text row under ITG the allowed are the queries and text . By the hypothesis those states depend only on text , the queries, and the image through the queries (claim 3).
The cross-attention reads query states and , and touches no text row. The FFNs and LayerNorms act on each row separately. So the claims hold after the layer. ∎
In plain words, under the generation mask the caption has to be written from the queries, because the image is not visible any other way. And the queries cannot cheat by reading the caption, because the mask hides it from them. That is what the paper means when it says ITG forces the queries to extract the visual features that capture the information in the text.
Claim 1 has a practical payoff. The query outputs under ITC and ITG are identical, so one image-side forward pass serves both losses. The released code runs the queries once and reuses their cached keys and values for the ITG decoder. An interviewer who asks how to make BLIP-2 pretraining cheaper is looking for this.
The cell checks all three claims on a tiny two-layer Q-Former with random weights. It changes the last word of the caption, or the image, and measures how much each output moves.
The zeros are exact, not small. Under ITC neither side moves when the other changes. Under ITG the queries ignore the caption, the text follows the image, and earlier words ignore later ones. Under ITM everything depends on everything.
The widget draws the three masks for queries and a caption of tokens. The strip on the right stands for the frozen image features, which the queries read through cross-attention.
A real batch adds one more rule. Captions have different lengths, so shorter ones are padded on the right, and no position may read a pad (§4.4). The pad rows still compute something, which is then ignored. But each row must keep at least one allowed key. Otherwise its softmax divides zero by zero and returns NaN.
Implement qformer_masks(m, text_valid). The argument text_valid is a boolean array of shape (B, T). It is True for real text tokens and False for right padding, and the first text token of every caption is real.
Return a dict with keys "itc", "itg", and "itm". Each value is a boolean array of shape (B, m + T, m + T) with the queries first. True means the row position may attend to the column position.
Apply the rules of the definition. Then block every padded text column in every row, and leave the rows themselves alone. Use no Python loops over positions.
The contrastive loss is the CLIP loss of §7.2 with one change. Here is that loss again. A batch holds image-caption pairs. Each image and each caption is encoded to a unit vector, and is the cosine between image and caption . Each image should pick its own caption out of the , and each caption its own image. So the loss averages two cross-entropies over the logits ,
In plain words, asks each image to pick its own caption, and asks each caption to pick its own image. The other items of the batch serve as the wrong answers, called negatives. The temperature is one number that sets how sharp the softmax is. The change for BLIP-2 is that an image now has 32 embeddings instead of one.
In plain words, to score a caption against an image, let every query compete and keep the one that agrees best. Different queries end up describing different things, and a caption usually talks about only some of them. A caption that says "a dog on a skateboard" should be scored by the query that captured the dog. It should not be scored by an average that mixes in the queries that captured the sky and the pavement. Here is the effect in numbers you can check by hand.
This way of scoring is called late interaction. Each side is encoded on its own, and the comparison takes the best match among several vectors instead of one dot product. Text retrieval uses the same idea, where each query word finds its best-matching passage word (Khattab and Zaharia 2020, §7.3). Here there are many vectors on the image side and one on the text side.
Late interaction also escapes a limit of single vectors. §30.1 showed that when every score is one dot product of -dimensional vectors, the table of all scores has rank at most . A maximum over 32 dot products is not one dot product, so its table of scores is not bound that way.
The maximum also shapes the gradient. Where the maximum is unique, , with the winning query. So
In plain words, each image-caption pair, positive or negative, sends its whole gradient to a single query. A query that wins no pair in the batch gets no contrastive gradient at all, though the other two losses still train it. Winner-take-all updates like this push the queries to divide the content of images between them.
BLIP-2 uses plain in-batch negatives. Its predecessor BLIP (Li et al. 2022) also kept a momentum queue, a store of embeddings from recent batches made by a slowly updated copy of the encoder, to get more negatives than one batch holds. BLIP-2's image encoder is frozen and runs in half precision, so many more samples fit on each GPU. Its stage-1 batches hold 2,320 pairs with ViT-L and 1,680 with ViT-g, and the batch itself supplies enough negatives. The released code gathers the embeddings from every GPU, so each image is contrasted with the captions of the whole batch. It also smooths the targets of the contrastive loss, putting 0.9 of the weight on the true partner and spreading 0.1 over all the candidates (label smoothing, §0.5).
How much does the maximum matter at scale? The next cell builds a toy world. Each image contains objects, its queries have each captured one of them, and each caption names a single object. The cell compares three ways to score a caption against an image. They are the maximum over queries, the mean of the query cosines, and the cosine with the mean query vector. The measure is text-to-image recall at 1, the fraction of captions whose best-scoring image in a batch of 256 is their own.
With one object per image the three scores coincide (96.9 percent). With 4 objects the maximum keeps 90.2 percent, while the two averages fall to 75.8 and 78.5. With 16 objects the averages collapse to 11.7 and 12.5 percent, while the maximum holds 67.6. The more an image contains, the more an average dilutes the one thing a caption mentions. Real queries are not this tidy, but the direction of the effect is the reason for the design.
Implement itc_loss(Zq, T_cls, tau). The array Zq has shape (B, m, D) and holds the query outputs of images. The array T_cls has shape (B, D) and holds the text embeddings of their captions, in the same order.
Normalize every vector to unit length. Form the logits S[i, j] , and return (loss, S). The loss is the average of two cross-entropies, image to text over the rows and text to image over the columns, with the diagonal as the positives. Use a stable log-softmax and no Python loops.
The generation loss is ordinary next-token prediction on the caption (§3.1), run under . The first text position holds [DEC] and predicts the first word. Each word then predicts the word after it, and pads carry no loss. The released code smooths these targets by 0.1 as well. The logits come from BERT's language-model head, whose output matrix is the word-embedding matrix reused.
Nothing in the loss is special. The mask does all the work, through claim 3 of the theorem. And the queries get a gradient from every caption word, because every word was predicted from them.
Does generation help even when all you want is retrieval? The paper's Table 6 says yes, by about a point. The authors fine-tuned the image encoder and the Q-Former for retrieval on COCO, a dataset of photos with five captions each. With ITC and ITM alone, BLIP-2 ranked the right caption first for 84.5 percent of images and the right image first for 67.2 percent of captions. Adding ITG raised these to 85.4 and 68.3. The authors read this as ITG pushing the queries toward features that matter for language. It is one ablation at one scale, an empirical finding.
ITM runs the Q-Former under . So the queries read the caption while they read the image, and their outputs mix both. A linear head with bias turns each query output into two logits, match and no match. The logits are averaged over the queries,
and trained with a two-class cross-entropy. In plain words, averaging the logits of a linear head is the same as applying the head to the mean query output.
The two heads split the work the way search engines do. ITC is a bi-encoder. Each side is encoded alone and compared by one dot product per query, so millions of images can be scored against a caption cheaply. ITM is a cross-encoder. The two sides read each other in every layer, which is more accurate and far more expensive. That is the split between retrieval and reranking in §7.5, and BLIP-2 uses its heads that way. It takes the 128 best candidates by ITC similarity and reranks them by ITM score.
A matching classifier trained on random negatives learns little, because a random caption is easy to reject. So the negatives are hard negatives, wrong captions that the contrastive scores find confusable. For each image, a wrong caption is sampled with probability proportional to over the captions that are not its own. Symmetrically, a wrong image is sampled for each caption. BLIP-2 took this strategy from ALBEF (Li et al. 2021) and BLIP.
The released code then scores pairs per step. There are positives, captions each paired with a hard negative image, and images each paired with a hard negative caption. The total stage-1 loss is with equal weights.
Two details decide whether this works, and interviewers ask about both. First, a caption that duplicates the image's own caption is a false negative, a pair labeled wrong that is actually right. COCO has five captions per image and web data repeats. A duplicate also has exactly the positive's score, which training pushes above every other, so the sampler would favor it most. So duplicates should be excluded along with the positive (§7.2 discusses false negatives). When image ids are available, as in COCO fine-tuning, the released code excludes every caption of the same image. Second, the probabilities must be renormalized after the exclusions so they sum to one.
Implement sample_hard_negatives(S, same, u, tau). The array S has shape (B, B), with S[i, j] the ITC similarity of image and caption . The boolean array same has shape (B, B). It is True where caption is a positive for image , which is always the diagonal plus any duplicate captions. The array u has shape (B,) with values in .
For each image , give caption probability proportional to when same[i, j] is False, and zero otherwise, renormalized to sum to one. Every row has at least one allowed caption. Return the smallest whose cumulative probability exceeds , as an integer array of shape (B,). Round-off can leave every cumulative probability of a row at or below a close to 1. Then return the last caption with nonzero probability, never an index out of range. The function must be stable for small .
After stage 1 the queries extract something useful from any image. Stage 2 connects them to a frozen large language model. The Q-Former now runs on the image alone, with no text. So its output is the text-independent of the theorem's claim 1. One fully connected layer maps it to the language model's width. The result is a soft prompt, a set of vectors placed in the input where token embeddings would go, but not tied to any word.
BLIP-2 tried two families of language models. OPT (Zhang et al. 2022) is a family of decoder-only models trained only on next-word prediction. FlanT5 (Chung et al. 2022) is the T5 encoder-decoder model further trained to follow instructions.
Prompt tuning (§14.6) learns one fixed soft prompt per task. These 32 vectors are instead computed from the image, a different prompt for every picture. The loss masking is that of §14.3, and the prefix language-modeling split is the objective of §8.1.
The shapes are concrete. OPT-6.7B has . So and its bias have parameters, and . OPT-2.7B has , FlanT5-XL , and FlanT5-XXL .
Why not skip stage 1 and train the Q-Former through the language model directly? Then it would be a resampler trained the Flamingo way, and the paper ran exactly that ablation.
The full recipe, from the paper's training details, is short. Pretraining uses 129 million images in total. Of these, 115 million come from LAION-400M and the rest from COCO, Visual Genome, CC3M, CC12M, and SBU. For each web image, BLIP's large captioner writes 10 synthetic captions. These are ranked together with the original web text by their CLIP ViT-L/14 similarity to the image. The top two are kept, and one is sampled at each step.
Stage 1 runs 250,000 steps and stage 2 runs 80,000. Both use AdamW, a peak learning rate of after 2,000 warmup steps, cosine decay (to no lower than in stage 2), and images. The frozen ViT and language model run in FP16 (BF16 for FlanT5).
Table 2 of the paper shows the trends. A larger image encoder helped. With OPT-2.7B, moving from ViT-L to ViT-g raised zero-shot VQAv2 from 49.7 to 52.3, and with FlanT5-XL from 62.3 to 63.0. A larger language model in the same family helped too, from 52.3 to 52.6 going from OPT-2.7B to OPT-6.7B, and from 63.0 to 65.0 going from FlanT5-XL to XXL. The instruction-tuned FlanT5 beat OPT, which had no instruction tuning, by about 10 points at similar size.
OK-VQA asks questions that need world knowledge more than vision. There Flamingo-80B stayed ahead, 50.6 against 45.9. The authors attribute that to Flamingo's 70-billion-parameter Chinchilla language model knowing more than the 11-billion-parameter FlanT5-XXL. In practice the frozen language model sets the ceiling on knowledge, and the connector cannot raise it.
Implement qformer_params(d_vis, d=768, d_ff=3072, L=12, m=32, V=30522, n_pos=512, cross_every=2), which returns (full, image_side), and stage2_trainable(d_vis, d_llm). Every linear layer has a bias, and every LayerNorm has a gain and a bias of width . Count these parts.
The image_side count includes only the parts that process queries. Those are the queries, the embedding LayerNorm, the self-attention blocks, the query FFNs, and the cross-attention blocks. The function stage2_trainable adds the FC layer to image_side.
§31.1 priced an image in a language model by its token count. Each visual token in the sequence costs about matrix-multiply FLOPs of prefill, the forward pass over the prompt, for an -parameter model (§5.6). It also costs cached numbers, a key and a value for each of layers and key-value heads of width (§6.5). The cell applies these formulas to a 7-billion-parameter model with layers and 32 key-value heads of width 128, the shape of OPT-6.7B and Vicuna-7B, in fp16.
BLIP-2's 32 vectors are the same whatever you ask about the image. That is the price of claim 1. The queries were trained to summarize the image for a caption, and a caption describes the main content. A question may ask about a small sign in the corner.
BLIP-2's own fine-tuning for question answering already fed the question into the Q-Former. The question's tokens joined the queries in self-attention, so they could steer the cross-attention toward the relevant regions. InstructBLIP (Dai et al. 2023) made this the architecture. The instruction text enters the Q-Former next to the queries, under a mask that lets them read each other. The query outputs, now aware of the instruction, go through the FC layer to a frozen language model (FlanT5 or Vicuna) that also reads the instruction.
InstructBLIP started from BLIP-2's pretrained weights and tuned only the Q-Former, with the image encoder and the language model frozen. BLIP-2 was never released with Vicuna, so for those models the authors first ran BLIP-2's two pretraining stages themselves. InstructBLIP gathered 26 public datasets, trained on 13 of them, and evaluated zero-shot on the other 13. Removing the instruction from the Q-Former hurt most where the question decides where to look. It cost the FlanT5-XL model 7.0 points on ScienceQA, which asks science questions about images. It cost the Vicuna-7B model 15.4 points on iVQA, a video question-answering dataset.
The cost is caching. Queries that ignore the instruction can be computed once per image and reused for any number of questions. Queries that read the instruction must be recomputed for every new question. An interviewer may ask which you would ship for a multi-turn chat about one image, and the answer turns on this trade.
Every compressor in this section, and Flamingo's resampler, ends with vectors where the encoder produced hundreds. A few vectors cannot hold everything. One theorem makes that exact for the simplest case, rebuilding the patch features linearly from the outputs.
Start with a picture. Suppose the patch features of an image vary in only three independent directions. Then three well-chosen summaries can rebuild every patch exactly, and a fourth adds nothing. Real features vary in many directions with different strengths. The strengths are measured by the singular values of the feature matrix (§0.10), and summaries can at best keep the strongest directions.
In plain words, rebuild each patch as a linear mix of the outputs. Then, whatever the queries attend to, the error is at least the energy of outside its top singular directions.
Be clear about what the theorem does not say. It does not cover a stack of nonlinear layers after the bottleneck, since an MLP applied to each rebuilt patch can raise the rank. And the language model does not need to rebuild the patches, only to answer questions. So the theorem is a precise version of an intuition. Few tokens lose detail first in images whose features have a slowly falling spectrum, such as dense text and many small objects.
It depends on which readouts we allow. If may be chosen per image, any linearly independent outputs reach the bound, because then ranges over every matrix of rank at most . But such a is computed from itself. It is an oracle that measures what the bottleneck permits, not what a decoder that sees only the outputs can do.
So fix the feature map and ask what the summaries hold. Take . This can only help the readout, because a projection either can be undone (when it has full row rank) or loses more. The most generous readout lets each patch pick its own mix of the summaries, with free. For each patch this is a least-squares problem, to find the combination of the rows of closest to . Its answer is the orthogonal projection of onto the row space of . So the best error is
with the pseudoinverse and the projector onto that row space.
Write . If the row space of is the span of the top right singular vectors , then , the truncated SVD itself, and the bound holds with equality. The choice does it, giving .
Attention weights must be nonnegative with rows summing to 1, which those rows are not. But suppose the constant vector lies in the span of . Then adding a large enough multiple of to each row and renormalizing makes the rows valid attention weights, and it keeps the row space inside that span. For a generic shift the shifted rows still span all of it.
The cell tests this on synthetic patch features with a known spectrum. A grid of patches gets features . The columns of are the 96 lowest-frequency 2D cosine patterns on the grid, so they are orthonormal, and is a random orthogonal matrix. The singular values are for a pattern of spatial frequency , so a larger makes a smoother image. The constant pattern is , so the shifted best queries of the derivation are valid attention weights.
The cell compares three designs with the bound. They are these best queries, average pooling over an grid of blocks with (as in MM1's ablation), and softmax attention with random scores.
The widget runs the same experiment for twelve values of between 1 and 64. It also rebuilds one feature channel from the summaries, so the lost energy shows as lost detail. The dashed line is the bound. Before you slide, choose pooling and the detailed image at , and predict whether the rebuilt channel looks like a blurred copy or like noise. Then switch to the smooth image without moving , and ask why the same pooling looks so much better.
The rank bound counts what a linear readout can rebuild, which is not the same as what the vectors could store. A language model reads the summaries through many nonlinear layers, so it is fair to ask how much information 32 vectors can hold at all.
Implement two functions, for Z of shape (N_p, d) and pooling weights A of shape (m, N_p). Both return Python floats, and neither centers Z.
bottleneck_bound(Z, m) returns the fraction of that lies outside the top singular directions, .pooled_reconstruction_error(Z, A) returns the smallest value of over all .The tests check the bound against known singular values, and that a pooled error never beats the bound. They also check that the error reaches the bound when the rows of span the top singular directions, and that duplicated queries add nothing.
The Q-Former is one point in a design space that every vision-language team explores. The table lays it out, with what each design keeps and what trains it. Two entries need a definition. A C-Abstractor is a small convolutional network, ResNet blocks with adaptive pooling, that shrinks the patch grid while keeping neighbors together. Pixel shuffle stacks each block of neighboring patches into one wider token.
| connector | visual tokens | keeps local layout | trained by | example |
|---|---|---|---|---|
| linear or MLP per patch | (576 at 336 px) | yes, one token per patch | the LM loss | LLaVA, LLaVA-1.5 (§31.1) |
| average pooling, then linear | coarsely, by blocks | the LM loss | MM1's ablation | |
| Perceiver resampler | fixed (64) | only through the features | the LM loss, through gated cross-attention or as prefix tokens | Flamingo, Idefics2 |
| one cross-attention layer with learned queries | fixed (256) | 2D positions added to queries and keys | the LM loss | Qwen-VL (Bai et al. 2023) |
| Q-Former | fixed (32) | only through the features | ITC, ITG, ITM first, then the LM loss | BLIP-2, InstructBLIP, MiniGPT-4 |
| C-Abstractor (ResNet blocks, adaptive pooling) | any square number | yes, convolutions | the LM loss | Honeybee, MM1 |
| pixel shuffle or 2x2 merging, then MLP | yes, neighbors merged | the LM loss | InternVL, Idefics3, Qwen2-VL (§31.5) |
The Q-Former's reuse shows its appeal. MiniGPT-4 (Zhu et al. 2023) took BLIP-2's frozen ViT-g and pretrained Q-Former and trained only one linear layer into a frozen Vicuna. The idea also travels beyond images. SALMONN (Tang et al. 2023) compresses speech with a Q-Former that emits one query output per window of about a third of a second (§35.6). Robot policies built on vision-language models face the same budget, with several cameras at many frames per second. Some compress each frame into a few learned tokens for the same reason, which vision-language-action models inherit.
What decides between the designs? Three controlled studies, each in its own setting, point the same way.
The two findings seem to disagree. MM1 says more tokens help, and Idefics2 says 64 are enough. The kind of task settles it. Idefics2's own authors, building Idefics3 a few months later (Laurençon et al. 2024b), replaced the perceiver resampler with pixel shuffle. Their reason was that 64 tokens per image were a bottleneck for reading text, so they encoded each tile into 169 tokens. Their survey adds that compression to 64 tokens holds up for most tasks, except those that need extensive text reading.
The rank bound suggests the same split. Every patch of a page of text carries different characters, so its features plausibly have a flatter spectrum than those of a photograph, where large regions look alike. A flat spectrum is exactly the case where a few summaries lose the most.
Locality is the other axis. Honeybee (Cha et al. 2024) observed that a resampler's queries tend to concentrate on a few salient regions. It measured the effect on six spatial-understanding tasks, such as positions, spatial relations, and object localization, with Vicuna-7B and CLIP ViT-L/14. At 144 tokens the resampler scored a normalized average of 43.9 and the convolutional C-Abstractor 53.5. The C-Abstractor even beat a linear projector with all 256 tokens (52.6). Learned queries see the image as a set, and a convolution sees neighborhoods.
The training tasks themselves did not survive scaling. BLIP-3 (Xue et al. 2024), the same lab's successor, replaced the Q-Former with a perceiver resampler, and its three losses with the single next-token loss on text. It cited the intricate Q-Former and its three objectives as obstacles to larger-scale training, and BLIP-2's single-image format as a limit. In BLIP-3's own ablation, the resampler with 128 queries beat an MLP with pooling on average over four groups of benchmarks. It lost on text reading, 70.4 against 72.2, and narrowly on hallucination.
BLIP-2's single-image format had a visible cost. Its paper saw no gain from adding worked examples to the prompt, because each training sample held one image-caption pair. Interleaved image-text data, the subject of §31.2, is what teaches that skill.
LLaVA-1.5 (Liu et al. 2023b) gave a reason from instruction following. In InstructBLIP only the Q-Former is trained. So its visual tokens must also steer the answer's format, short or long, the way a soft prompt does, and a small module may lack the capacity. LLaVA-1.5 instead trained the language model and controlled the format with a short instruction in the prompt.
So the field converged on the simplest connector, an MLP over merged patches, with as many tokens as the budget allows. It spends its effort on resolution (§31.5) and on training the language model (§31.4). Learned-query compressors remain the tool when the budget is the binding constraint, with many images, long video, or a language model that must stay frozen.
| choice | when it wins | what it costs |
|---|---|---|
| Q-Former with stage 1 | frozen LLM, small training budget, fixed tokens per image, cheap retrieval and matching heads for free | its own pretraining with three losses, 32 tokens that lose dense text and spatial detail |
| perceiver or attention pooling trained by the LM loss | many images or frames per prompt, strict token budgets | the LM loss must teach what to keep, weaker locality |
| MLP projector over (merged) patches | strong LLM that can be unfrozen, OCR, documents, grounding | hundreds to thousands of tokens per image in every layer and in the cache |
| convolutional abstractor | flexible token count with locality kept | a design with more hyperparameters, little gain over pooling in MM1's setting |
The block builds a small Q-Former with PyTorch's fused attention. It takes a boolean mask in which True means the position may attend, the convention of §4.4. The block checks claim 1 of the theorem against a pass with no text at all. Then it computes the ITC loss with the maximum over queries, and shows that each image's gradient reaches only the queries that won some pair.
The two counts agree image by image. A query that won none of an image's comparisons received exactly zero contrastive gradient, as the derivation above said. To use the real model, Hugging Face's Blip2ForConditionalGeneration and InstructBlipForConditionalGeneration load the released weights.