This section is about the training data of a joint text-image model, the image-caption pairs it learns from. A model like CLIP learns to place a photo and its caption close together in one shared space (§30.1), so it can only learn what its pairs show. Those pairs come from the web, and most of them are noisy. Which pairs you keep changes the model's accuracy by more than most changes to its architecture or its loss, and choosing them well is now a field of its own.

Here is the evidence. Take one model and one training budget, and change nothing but the training set. The model is CLIP ViT-L/14, a CLIP whose image tower is a large Vision Transformer that cuts each image into 14-by-14-pixel patches (§27.2). The budget is 12.8 billion samples seen, the number of image-caption pairs the model processes in training, repeats included. The score is zero-shot ImageNet accuracy. Each of ImageNet's 1,000 classes is described by a sentence such as "a photo of a fox". The model labels each test image with the class whose sentence embeds closest to it, with no training on ImageNet (§30.3). Two studies ran exactly this experiment, DataComp (Gadre et al. 2023) and the data filtering networks study (Fang et al. 2023).

DataComp reports this budget as $1.1\times10^{21}$ multiply-accumulates, which matches one forward pass of both towers per sample. Two FLOPs per multiply-accumulate, times three for the forward and backward passes together, give about $6.6\times10^{21}$ training FLOPs. That matches the $6.7\times10^{21}$ counted from the towers' parameters in §30.2.

The largest set is not the best. OpenAI's 400 million pairs beat LAION-2B, almost six times larger, by 2.4 points. DataComp-1B, also smaller than LAION-2B, beats it by 6.1. So what separates these rows is which pairs were kept, not how many.

The section follows the methods that produced the table. We start with where the pairs come from and why their text is noisy. Then come four ways to choose pairs. You can keep the pairs that a trained CLIP model says match (LAION). You can build a benchmark that holds everything but the data fixed (DataComp). You can train a model whose only job is filtering (DFN). And you can balance how often each concept appears (MetaCLIP), which we prove is the best way to spend a fixed budget under a measured law. Four results then complicate the picture. The best filter depends on your compute. The best batch depends on what the model has already learned, so it is chosen during training. Synthetic captions help retrieval and can hurt classification. And web images carry legal and safety risks that no accuracy number shows.

Where the pairs come from

Every open image-text dataset starts from the same raw material. An HTML image tag can carry an alt attribute, a short text meant for screen readers and for browsers that cannot show the image. Common Crawl is a nonprofit that crawls the web and publishes the pages it finds. Parsing the image tags in its pages gives tens of billions of candidate pairs, each an image URL and its alt-text. Downloading the URLs that still work gives the images. The text side of this pipeline (extracting text, identifying languages, quality filters, removing personal data and toxic text, licensing) is the subject of §8.2. Here we care about what is special to pairs.

Alt-text is written for many purposes, and describing the picture is only one of them. In a random sample you will find

• nothing useful, such as camera filenames (IMG_2041.jpg), dimensions (1920x1080), or "image";
• generic words, such as "photo", "untitled photo", or "thumbnail";
• spam and boilerplate, product titles and keywords written for search engines ("buy cheap shoes photo sale");
• misaligned text, a caption about the article rather than the picture, or a person's name with no hint of what they are doing;
• and, sometimes, a real description ("a red fox in snow").

Two measurements show how much of the web is the first four kinds. YFCC100M is a public set of 100 million Flickr photos with their titles and descriptions. When CLIP's authors kept only the photos with natural-language English text, 100 million shrank sixfold, to 15 million (Radford et al. 2021). And Udandarao et al. (2024a) detected concepts in each image and in its caption separately, then counted pairs whose two sides share no concept. That was 5.3 percent of CLIP-filtered LAION-400M and 36.5 percent of YFCC-15M, with three other public datasets in between.

Every method in this section answers the same question, which pairs to keep. The early answers were hand-written rules, and they show the basic trade-off. Conceptual Captions, or CC3M (Sharma et al. 2018), filtered images, texts, and their agreement strictly and kept 3.3 million pairs. It also rewrote the survivors, replacing names with general words ("Harrison Ford" became "actor") and stripping numbers. Conceptual 12M, or CC12M (Changpinyo et al. 2021), relaxed the filters, rewrote only person names (into a placeholder token), and kept 12.4 million pairs.

The trade is precision, the share of kept pairs that are good, against recall, how much of the good material is kept. Human raters judged 90.3 percent of CC3M captions a good fit for their image, against about 76.6 percent for CC12M, though the two papers rated in different ways. But CC12M covers far more rare concepts. The word "mangosteen" occurs 212 times in CC12M and never in CC3M. ALIGN (Jia et al. 2021) went to the extreme of recall. It kept 1.8 billion pairs with a few frequency rules, and showed that scale alone can absorb a lot of noise. Its rules dropped alt-texts shared by more than 10 images, texts with rare tokens, texts shorter than 3 or longer than 20 words, and images with more than 1,000 alt-texts.

In practice, a strict filter gives clean data with a thin long tail, and a loose one gives a rich tail full of noise. Every later method tries to get both.

Curation by query, the WIT recipe

CLIP's own dataset, WIT (WebImageText), was built by search rather than by filtering. Its authors made a list of 500,000 queries and collected pairs whose text contains one of them. Then they balanced the result, keeping at most 20,000 pairs per query, so that no query could dominate (Radford et al. 2021). The queries came from four sources. The first was every word that occurs at least 100 times in English Wikipedia. The second was two-word phrases whose words occur together far more often than chance. The third was titles of Wikipedia articles above a search-volume threshold, and the fourth was WordNet synsets (sets of synonyms naming one concept) not already on the list. The resulting 400 million pairs hold about as many words as WebText, the text GPT-2 was trained on.

That paragraph is nearly all OpenAI published, and WIT was never released. The open efforts that followed took two routes. LAION replaced the query step with a filter built from the released CLIP model itself. MetaCLIP, years later, rebuilt the query recipe and found that its balancing step, the "at most 20,000 per query", was what mattered. We take them in that order.

Filtering by CLIP score

A trained CLIP gives a cheap test of whether a caption fits its image. Recall the model from §30.1. It has an image tower $f$ and a text tower $g$, each mapping its input to a vector in the same space. Throughout this section $x$ is an image, $y$ is its text, and the unit embeddings are $\hat{\mathbf u}(x) = f(x)/\lVert f(x)\rVert$ and $\hat{\mathbf v}(y) = g(y)/\lVert g(y)\rVert$. The cosine $\hat{\mathbf u}(x)^\top\hat{\mathbf v}(y)$ is high when the caption describes the image. So embed both sides of every pair, and keep the pairs with high cosines.

In plain words, ask a CLIP model how well each caption matches its own image, and keep the best-matching pairs. Two details matter in practice. Only the $N$ matched pairs are scored, never the $N\times N$ matrix of every image against every caption. And CLIPScore, the metric for generated images in §34.6, is a different number. It is this cosine clipped at 0 and multiplied by 2.5 (Hessel et al. 2021), so check which of the two a paper reports.

LAION-400M (Schuhmann et al. 2021) was the first large open dataset built this way. It parsed Common Crawl's image tags, dropped alt-texts under 5 characters and images under 5 KB, and removed duplicates by URL and alt-text. Then it kept the pairs whose cosine under OpenAI's CLIP was at least 0.3, a threshold chosen by looking at examples. LAION-5B (Schuhmann et al. 2022) did the same at a larger scale. It scored English text with OpenAI's CLIP ViT-B/32 and other languages with a multilingual ViT-B/32, at thresholds of 0.28 and 0.26.

Eight pairs, by hand

Before we criticize the filter, run it on something small enough to read. The pool below has eight pairs with made-up CLIP scores. We will also run a second filter on it, a preview of MetaCLIP's balancing, so each caption also lists which words of a three-word list ("photo", "fox", "lighthouse") it contains. Balancing works on those words only. It caps how many pairs each word may select. A word with more pairs than the cap selects each of its pairs at random, with probability cap divided by count. A pair is kept if any of its words selects it.

The two filters agree on the good pairs and disagree on the generic ones, for different reasons. CLIP scoring looks at the image and the text together. It rejects "buy cheap shoes photo sale" because the text does not describe the picture. Balancing never looks at the image. It keeps a few generic "photo" pairs, at random, because it only cares that no word dominates. Here is the same computation in code.

The last line previews a subtlety. The cap is 2, yet 3.2 pairs containing "photo" survive on average, because two of them are kept through their other words. We will prove exactly how many survive once balancing is defined.

What filtering by CLIP imports

The filter is a CLIP model, and that is both its strength and its flaw. It is cheap, needs no labels, and removes most of the junk in the list above. But it keeps the data that CLIP already finds matching. Whatever CLIP gets wrong, the next model is trained to get wrong too. Three problems follow.

• Text in the image. CLIP can read. An image that shows its own caption as printed text (a book cover, a product shot with the brand name, a meme) scores high whether or not the picture shows anything else. Maini et al. (2023) found that nearly 40 percent of LAION's images contain text that overlaps substantially with the caption. Training on them pays the next model to read text instead of learning what things look like. Their fix, T-MARS, masks the text in each image before rescoring it.
• Blind spots and biases. A concept the scorer knows badly, plausibly one it rarely saw, gets low cosines even when the caption is right, so its pairs are removed. The scorer's social biases pass into the data too, a point the LAION-5B authors make about their own dataset.
• A ceiling. LAION-400M, filtered by OpenAI's CLIP, trains a ViT-L/14 that is 2.7 points worse than one trained on OpenAI's WIT at the same compute (72.8 against 75.5). Almost six times more LAION data (LAION-2B) closes only 0.3 of those points. A filter built from a model copies that model's idea of a good pair, and a student rarely beats its teacher when the teacher also grades the homework.

The scorer also sets the scale of the numbers. In CommonPool, the top 30 percent of pairs begins at a cosine of 0.281 under CLIP ViT-B/32 but at 0.243 under ViT-L/14, and the top 10 percent at 0.325 and 0.295 (Gadre et al. 2023). The same threshold is a different filter under each model. That is why DataComp specifies its filters by a kept fraction, and its best ones use the ViT-L/14 scorer.

In practice, report a CLIP-score filter as a scorer plus a kept fraction, and check what it removes, not only what it keeps. Interviewers like to ask what goes wrong when you filter the data with the model family you are about to train.

The widget below runs a CLIP-score filter on a model of LAION-5B's 50 billion candidates.

Move the threshold from 0.20 to 0.35 and watch two things. The number of pairs falls smoothly. The share of concepts that keep at least 100 pairs, in the right panel, holds near 100 percent and then falls quickly. A rare concept has few pairs to begin with, and if the scorer knows it badly, those few pairs score low. Then turn on a balancing cap, and notice that it removes pairs without touching coverage.

import numpy as np

def clip_score_filter(U, V, threshold=None, keep_frac=None):
    raise NotImplementedError

DataComp, a benchmark where only the data changes

Before 2023, comparing datasets was nearly impossible. Each paper trained a different model for a different number of steps and evaluated it on a different set of tasks. DataComp (Gadre et al. 2023) turned the usual benchmark around. A normal benchmark fixes the data and lets you change the model. DataComp fixes the model, the training code, the hyperparameters, the compute, and the evaluation, and lets you choose only the data.

Why fix samples seen rather than the dataset's size? Because the size is the most important decision, how hard to filter, and fixing it would forbid that decision. Fixing compute makes the trade-off explicit instead. A set of half the pool is seen twice, a quarter of the pool four times. The benchmark asks whether the smaller, cleaner set is worth the repetition.

So is the smaller, cleaner set worth the repetition? At each scale tested, the answer is yes, up to a point. Random subsets of the pool are always worse the smaller they are, so less data is not good in itself. CLIP-score filtering does best near the top 30 percent, so less data selected well beats more data. The ranking of filtering methods was largely the same across scales, with rank correlations of 0.71 to 0.90 between scales, so cheap small-scale experiments predict the expensive ones. And as a check on the pool itself, LAION-2B's recipe applied to CommonPool at the large scale, at matched size, gave 55.3 percent on ImageNet against 55.7 for LAION-2B, so the two pools are comparable.

The best baseline adds an image-based filter to the CLIP-score filter. It takes the English pairs with long-enough captions, clusters their ViT-L/14 image embeddings into 100,000 groups, and keeps the groups nearest to some ImageNet training image. Intersecting that with the top 30 percent by CLIP score gives DataComp-1B.

Notice what the image-based filter uses, the ImageNet training images. That is not contamination, since the evaluation images were removed. But it selects data that looks like the evaluation's own distribution, and part of DataComp-1B's ImageNet gain comes from that. Whenever a dataset beats a benchmark, ask what its filter was told about the benchmark.

Learning the filter

If a CLIP model is a filter, is a better CLIP model a better filter? Fang et al. (2023) asked this and found that accuracy is the wrong measure of a filter. A model's ImageNet accuracy does not predict how good a dataset it selects. A filter with 30 percent lower ImageNet accuracy than OpenAI's CLIP could be as good a filter. What mattered was the data the filter itself was trained on. They trained a filter on a high-quality set, then replaced a growing share of that set with unfiltered Common Crawl pairs and trained again. The filter's own ImageNet accuracy declined only gradually, but the datasets it selected quickly became barely better than no filtering.

Their best data filtering network (DFN) is a ViT-B/32 CLIP. It starts from OpenAI's weights, trains on 357 million high-quality pairs, and is then fine-tuned on the training sets of COCO, Flickr30k, and ImageNet, with ImageNet labels turned into captions by templates. Keeping the top 15 percent of CommonPool by its score gives DFN-2B. On it, a ViT-L/14 reaches 81.4 percent at DataComp's xlarge compute, 2.2 points above DataComp-1B. The same filter applied to a 42-billion-pair pool gives DFN-5B. A ViT-H/14 trained on it, for 39 billion samples at 224 pixels and 5 billion more at 378 pixels, reaches 84.4 percent.

Two more findings change how you would spend clean data.

• High-quality pairs were worth more as training data for the filter than as training data for the final model. Adding the 357 million pairs to a filtered set helped less than using them to train the filter.
• Fine-tuning the filter on ImageNet improved the final model on ImageNet and on all its shifted versions. Adding ImageNet directly to the training set improved ImageNet and ImageNet-V2, but not ObjectNet, ImageNet-Sketch, or ImageNet-R. A filter pulls the dataset toward a target while keeping the variety of the pool it draws from.

In practice, if you have a small clean set, the best use of it may be to train your filter.

Balancing by metadata, the MetaCLIP recipe

LAION and DataComp asked a model which pairs were good. Xu et al. (2024) asked a different question, what OpenAI actually did. Their MetaCLIP rebuilds WIT's recipe from the one paragraph CLIP published, with no CLIP model anywhere in the loop. It has four steps, and the last one matters most.

Metadata. The 500,000 queries become a list $\mathcal M$ of metadata entries, the words and phrases a caption can be matched against. MetaCLIP rebuilt them from the same four sources.

The bigram rule uses pointwise mutual information, a measure of how much more often two words occur side by side than chance predicts. CLIP published neither of the last two thresholds. MetaCLIP chose the bigram threshold to give about 100,000 bigrams, then filled the list to 500,000 with the most viewed Wikipedia titles. It used page views because Wikipedia's search volume is not public.

Matching. Each text is matched against every entry by substring search. Texts that match nothing, such as dates, IDs, and filenames, are discarded. In a pool built from 15 Common Crawl snapshots, about half of the English pairs matched. That left 1.6 billion pairs with 5.6 billion matches, about 3.5 per text.

Counting. The count of an entry is how many pairs match it, and the counts are extremely uneven. The few entries with huge counts are the head, and the many entries with small counts are the tail. Counts like these follow roughly a Zipf law, where the entry of rank $r$ has a count proportional to $r^{-s}$ for some exponent $s$ near 1.

Balancing. Cap every entry at $t_{\text{bal}} = 20{,}000$ pairs. An entry at or below the cap is in the tail and keeps all its pairs. An entry above it is in the head and is subsampled to the cap. MetaCLIP does this in one pass over the pool with independent coin flips, so it never needs an index from entries to pairs.

In plain words, each entry votes for each of its pairs with probability cap over count, and one vote keeps a pair. A tail entry has $n_m \le t_{\text{bal}}$, so $p_m = 1$, and any pair that matches even one tail entry is kept for sure. An entry far into the head has a tiny $p_m$. MetaCLIP's paper puts the effect on "photo" vividly, saying that only 20,000 of its 54 million matches are kept. That is the number of pairs the entry "photo" selects. Pairs that contain "photo" and also some rarer entry are kept anyway, through that entry, as the toy pool showed. The next result makes this exact.

In plain words, the cap is a floor, not a ceiling. Every entry ends up with at least $\min(n_m, t_{\text{bal}})$ pairs. A head entry ends up with more whenever its pairs also mention other things, which for stopwords is nearly always. What the cap really bounds is how many pairs are selected because of the head entry. So texts that match many entries are favored. MetaCLIP intends this, since such texts tend to be denser in information. Stopping at the first coin that comes up heads, as MetaCLIP's pseudocode does, changes nothing, because the pair is already kept. Let us check the theorem by simulation on the toy pool.

The simulated keep rates agree with $q_i$ to within sampling error. The "photo" entry ends up with about 3.2 pairs, above its floor of $\min(5, 2) = 2$, exactly as part 4 says.

The widget applies the cap to MetaCLIP's pool, on a curve fitted to its published counts per entry.

The widget starts at CLIP's cap of 20,000. Before the cap, the 16,000 head entries hold about 95 percent of the pool's matches. After it, about half of what the entries select belongs to the tail. Drag the cap down and less survives, under 2 percent of the matches at a cap of 1,000. To curate a larger pool, MetaCLIP raised the cap with it. On 10.7 billion matched pairs a cap of 170,000 kept 2.5 billion, with the same 6 percent share of matches on tail entries as the 400 million pairs curated at 20,000.

What balancing bought

Every MetaCLIP comparison trains with CLIP's own recipe, a batch of 32,768 and 12.8 billion samples seen, so only the data differs. (The WIT row is OpenAI's released model, which was trained with that same recipe.) The ablation on ViT-B/32 separates the steps.

Matching alone gains 3.4 points over raw English pairs, and balancing gains 4.7 more. Training on four times more data, the whole unbalanced matched pool, is 3.6 points worse than the balanced 400 million. Among caps of 15,000, 20,000, and 35,000, CLIP's 20,000 tied 15,000 on ImageNet and was best on the average over 26 tasks.

import numpy as np

def entry_counts(matches, n_entries):
    raise NotImplementedError

def expected_kept(matches, n_entries, t_bal):
    raise NotImplementedError

Why flattening the head helps

Balancing throws away more than 70 percent of a pool that is already filtered, and the model gets better. Why? The plain answer is that the head is redundant. The 54 millionth "photo" pair teaches the model almost nothing the first million did not. A pair about a mangosteen may be one of a few hundred the model will ever see.

The same logic set the language weights of multilingual pretraining in §8.4. There each language is sampled in proportion to a power of its share of the data, with an exponent below 1, which gives small languages more weight than their size. (§8.4 calls that exponent $\alpha$. This chapter reserves $\alpha$ for the log of CLIP's logit scale, so here we write the exponent out in words.) A cap is a sharper version of the same idea. It samples in proportion to size below $t_{\text{bal}}$ and equally above it.

A measurement turns "the head is redundant" into a law. Udandarao et al. (2024a) took 4,029 concepts from 27 downstream tasks. They estimated how often each concept appears in five pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-Aesthetics, and LAION-400M) by tagging both the captions and the images. Then they compared each concept's frequency with how well 10 CLIP models and 24 text-to-image models handle it.

This is why "zero-shot" is a misleading word for CLIP (§30.1, and the audit in §30.3). It means the model never trained on the test dataset, not that it never saw the concepts. Accuracy on a concept is mostly a function of how much of it the model saw. And a log-linear law has a sharp consequence for how to spread a fixed budget of pairs across concepts.

In plain words, the value of one more pair of concept $m$ is proportional to $1/(1 + k_m)$, the slope of the logarithm, so the next pair should always go to the concept with the fewest pairs so far. Fill every concept up to a common level, and stop filling the ones that run out. This allocation is called water-filling. §8.4 uses the same rule for languages, giving every language an equal share of the training budget up to a limit on how often its data repeats (Chung et al. 2023). MetaCLIP's cap is the same answer for concepts.

The assumptions are strong. They are one concept per pair, equal slopes, no transfer between concepts, and equal weight per concept in the evaluation. So read the theorem as the reason a cap is the right shape, not as a proof that 20,000 is the right number. If the evaluation weights concept $m$ by $w_m$, the same argument gives $k_m = \min\big(n_m, \max(0, w_m/\lambda - 1)\big)$ for a common multiplier $\lambda$. That is roughly proportional to $w_m$ when the levels are large, and zero for concepts whose weight is too small.

The cell compares three ways to spend 600,000 pairs from a Zipf pool of 3.3 million. The first is proportional sampling, a random subset. The second is temperature sampling with exponent 0.5, each concept weighted by the square root of its count. The third is the cap. Each allocation is scaled to the same total.

The cap has the highest average of $\log_{10}(1 + k_m)$, as the theorem promises, 1.695 against 1.661 for temperature sampling and 1.085 for a random subset. In accuracy terms, with a slope of $b$ points per tenfold increase in frequency, moving from a random subset to a cap is worth about $0.61\,b$ points on the average concept. Temperature sampling gets most of the way there. A random subset spends 64.9 percent of the budget on the top 1 percent of concepts.

Balancing in a small simulation

The theorem assumed a law. Now let us watch the effect come out of training instead. The world below has 100 concepts with Zipf frequencies. An image is a noisy linear view of its concept's hidden vector. A caption is a bag of words, the concept's word plus two random filler words. We train CLIP with linear towers and the symmetric loss, as in §30.2. Its gradient with respect to the $B\times B$ logits is $(\mathbf A + \bar{\mathbf A} - 2\mathbf I)/(2B)$. Here $\mathbf A$ is the softmax of the logits along each row (each image over the captions), and $\bar{\mathbf A}$ is the softmax along each column (each caption over the images). Then we classify held-out images zero-shot, with each concept's word as its prompt.

The pool has 40,000 pairs. One training set is MetaCLIP's balancing with a cap of 60. The other is a random subset of the same size. So the two models see the same number of distinct pairs and train for the same 300 steps.

Balancing lifts the tail from 0.496 to 0.707, the coverage effect the theorem predicts. It also lifts the head, from 0.672 to 0.727, which no frequency law explains. The second run says why. In the random subset, 31 percent of pairs belong to one concept, so a batch of 128 holds about 40 copies of it. The contrastive loss then pushes each of those images away from 39 captions that describe it equally well. These are in-batch false negatives, other pairs in the batch that are treated as wrong answers although they match, and they are a cost specific to contrastive training.

Masking them restores the random model's head to 0.815 but leaves its tail at 0.500, so the two effects are separate. Balancing fixes both at once. It fixes the tail by coverage, and the head by making duplicates rare within a batch.

Real pools add a third reason, measured rather than simulated. CLIP's unusual robustness to distribution shift comes mostly from the variety of its training data, not from language or from the loss (Fang et al. 2022, discussed in §30.3). Flattening the head is a direct way to buy variety.

In an interview, "why does capping frequent concepts help?" has three answers, and a strong candidate gives all three. Tail pairs are worth more under a log-linear law, fewer duplicates means fewer false negatives in each batch, and more variety means more robustness.

How much filtering your compute can afford

DataComp found that keeping about 30 percent was best, with compute fixed at one pass over the pool. Goyal et al. (2024) asked what happens when compute changes, and found that the answer moves. Their pool was DataComp's 128-million-pair medium pool, and LAION-style filtering keeps about 10 percent of it. At 128 million samples seen, that filter beats no filtering by 7.5 points averaged over 18 tasks. But by about 450 million samples seen, each kept pair has been repeated over 30 times, and training on the unfiltered pool is better. Among CLIP L/14 filters that keep the top 10 to 20, 30, or 40 percent, the hardest wins at 32 million samples seen and the gentlest at 640 million.

The mechanism is the one §9.4 measured for text. Repeated data is worth less each time it is seen. High-quality data is scarce, so a hard filter forces many repetitions. At some budget, a fresh lower-quality pair is worth more than the twentieth view of a good one. Goyal et al. model this with a value per sample that halves after a fitted number of repetitions, one half-life per quality bucket. We will use a much simpler toy with the same shape.

In plain words, a harder filter raises the average quality of what you keep and lowers what each view is worth. The best filter balances the two.

Here is a tiny example. Two examples have qualities 1 and 0.5, both are kept, $S = 5$, and the value halves every view ($h = 1$). Then $P = 2$ full passes and $R = 1$ sample of a third pass. The total quality is 1.5. The first view of each is worth 1, the second $1/2$, and the third $1/4$ with probability $1/2$. So $U(2) = 1.5 \times (1 + 0.5 + 0.5 \times 0.25) = 2.4375$. Keeping only the best one instead gives $1 \times (1 + 0.5 + 0.25 + 0.125 + 0.0625) \approx 1.94$, less, so here it pays to keep both.

Two limits check the model. If repeats are worthless ($h \to 0$), each example counts once, so you keep exactly as many as you can see once, $K^\star = \min(S, N)$. If repeats are as good as new ($h \to \infty$), then $U(K) = (S/K)\sum_{i\le K} \omega_i$ is $S$ times the average quality of the kept set. That is largest at $K = 1$, so you would train on the single best example forever. Real data sits between the two, and the optimum moves toward keeping more as $S$ grows. This is our toy, not the fitted law of Goyal et al.

import numpy as np

def toy_utility(q_sorted, K, S, h):
    raise NotImplementedError

def best_keep_fraction(quality, budget, h):
    raise NotImplementedError

Here is the toy for a pool of 1,000 examples, vectorized over $K$, at three half-lives. The qualities are spread like a Beta(2, 5) distribution. They are taken as its quantiles rather than sampled, so the pool has no sampling noise, and the widget below uses exactly the same pool.

At every half-life, the best fraction never falls as the budget grows. And the more slowly repeats lose value, the harder you should filter at a given budget. For a real team the lesson is procedural. Tune the filter at the compute you will actually train with, or fit how its value decays with repetition. A threshold tuned in a cheap pilot run is tuned for too little compute. Interviewers ask exactly this, why the best filtering threshold depends on the budget.

The plot shows only where each optimum lands. The widget shows the whole curve that produces it, for one budget and one half-life at a time.

At the default, 1,000 samples seen and a half-life of 4 views, the best plan keeps the top third ($K^\star = 333$). It sees each kept example three times and gets 40 percent more utility than training on the whole pool once. Drag the budget down to 100 and the optimum jumps to the top 5 percent, each example seen twice, at more than twice the utility of no filtering. Drag it up to 10,000 and the best plan keeps 89 percent and beats no filtering by under 2 percent, because the clean examples have already been seen eleven times each. Then hold the budget at 1,000 and lengthen the half-life to 16. The optimum moves left to the top 12.5 percent, seen eight times each, since repeats now cost little. The solid curve is the value of one sample seen, $U(K)/S$. The dashed curve is the average quality of the kept set, the limit where repeats are as good as new. The value axis is logarithmic, so the gap between the two curves is the factor that repetition costs.

Choosing each batch as you train

Every filter so far is offline. A pair gets one score, computed once by a fixed model before training starts, and training then draws uniformly from what was kept. Such a score cannot know two things.

• It cannot know what the model being trained has already learned. A pair that was informative at step 1,000 is drawn just as often at step 100,000, long after the model has absorbed it. That is the toy's decay with repetition, now example by example.
• It cannot know which other pairs will share the batch. A contrastive loss scores every pair against the negatives beside it (§30.2), so a pair's value depends on its batch.

Online batch selection answers both. At every step, draw a super-batch $\mathcal S$ of $B/\phi$ candidates uniformly from the pool. Score them, and take the optimizer step on only the $B$ that score best. A fraction $\phi$ of each super-batch is kept, and the papers below call $1 - \phi$ the filtering ratio. The question is what to score.

Two models are involved. The learner is the model being trained, with current parameters $\boldsymbol\theta$. The reference model is fixed, with parameters $\boldsymbol\theta_{\text{ref}}$, and was pretrained on a small, carefully curated set. Write $\ell(x\mid\boldsymbol\theta)$ for the loss of a candidate pair $x$ under parameters $\boldsymbol\theta$. Three scores suggest themselves.

• Hard for the learner, a high $\ell(x\mid\boldsymbol\theta)$. It skips what the model already knows. But on web data the pairs with the highest loss are the mismatched ones, such as a caption about the article or a camera filename, which no model can learn. They stay hard forever, so this score fills the batch with noise. It helps on small, clean datasets and hurts on large, uncurated ones, as Evans et al. (2024) summarize the earlier studies.
• Easy for the reference, a low $\ell(x\mid\boldsymbol\theta_{\text{ref}})$. This is CLIP-score filtering again, applied online. It removes the noise, but it ignores the learner and keeps choosing pairs the learner learned long ago.
• The difference of the two, which avoids both failures at once.

In plain words, a pair is worth a training step when the learner still gets it wrong and a model trained on clean data gets it right. The four corners of the plane of the two losses make the comparison concrete.

The difference is more than a good heuristic. Mindermann et al. (2022) asked which candidate would most improve the model's predictions on a clean held-out set if the model trained on it next. Their answer, the reducible holdout loss, is this difference, derived below.

The derivation also says what the reference is for. It stands in for "a model that has seen clean data", so its training set defines what counts as worth learning, as the filter's training data did for DFN. Online selection also builds the toy's half-life in. Once the learner trains on a pair, its loss on that pair falls toward the reference's loss. The pair's learnability falls, and it is chosen less often, without anyone fitting $h$.

The widget places a made-up super-batch of candidates by their two losses and shows which ones each score keeps.

Start with "hard for the learner" and read the bar under the plot. Much of the kept batch is noise. Switch to "easy for the reference" and the noise disappears, but most of the batch is pairs the learner already gets right. Dragging training progress to the right only lowers the learnability of what it keeps. Switch to learnability and the batch is the learnable pairs alone. Now press Train. The learnable points sink toward the diagonal as the learner catches up. Near the end, the selection starts to admit already-learned pairs and a little noise, a sign that this super-batch is used up and fresh candidates are needed. The points outside the reference's data are never chosen, which is the price of steering by a reference.

Why the batch must be chosen jointly

The scores so far treat each candidate on its own, and a contrastive loss does not. Take the sigmoid loss of §30.4, which treats every image-text pair in the batch as its own yes-or-no question. For a batch $\mathcal B$ of $B$ pairs with unit embeddings $\hat{\mathbf u}_i$ and $\hat{\mathbf v}_j$, the logits are $s_{ij} = e^{\alpha}\hat{\mathbf u}_i^\top\hat{\mathbf v}_j + b_0$, with $\alpha$ the learned log scale and $b_0$ the learned bias. The labels are $z_{ij} = 2\,\mathbb 1[i=j] - 1$, that is $+1$ for a matching pair and $-1$ otherwise. Write the term for image $i$ and text $j$ as

Pair $i$'s share of the loss, $\ell_{ii} + \sum_{j\ne i}\ell_{ij}$, depends on which captions $j$ sit in the batch with it, and so does its learnability. Compute every term under the learner and under the reference, each with its own $\alpha$ and $b_0$. The differences form the learnability matrix of the super-batch, $\boldsymbol\Delta\in\R^{\lvert\mathcal S\rvert\times\lvert\mathcal S\rvert}$, with entries $\Delta_{ij} = \ell_{ij}(\boldsymbol\theta) - \ell_{ij}(\boldsymbol\theta_{\text{ref}})$. It holds everything selection needs.

In plain words, selecting examples one at a time sees only the diagonal of $\boldsymbol\Delta$, the matching pairs. The off-diagonal entries are the negatives. $\Delta_{ij}$ is large when the learner still confuses image $i$ with caption $j$ while the reference tells them apart, say a fox photo that the learner keeps matching to "a grey wolf in the snow". Two such pairs are worth more together than either is alone, because each supplies the other with a learnable hard negative. That generalizes the hard-negative mining of §7.2, mining positives and negatives at once (Evans et al. 2024). Part 3 rules out the exact answer at scale. Part 2 gives the natural heuristic, which is to grow the batch a piece at a time and score each candidate by what it adds to what is already in.

Joint example selection, or JEST (Evans et al. 2024), is part 2 with sampling instead of a strict top-k. It fills a sub-batch of 32,768 pairs in $N = 16$ chunks of 2,048. The first chunk is sampled without replacement, with probabilities proportional to the exponential of a scaled diagonal learnability $\Delta_{ii}$. Each later chunk is sampled the same way, using the conditional score of part 2 given every chunk already chosen. Within a chunk, candidates are drawn independently of one another, so the 16 chunks need 16 sequential rounds instead of 32,768. In the paper's measurements, 16 chunks reached sub-batches as learnable as a slow search that swaps candidates in and out one at a time (Gibbs sampling) for thousands of iterations. The exercise builds the deterministic version, which takes the top scores instead of sampling them.

import numpy as np

def select_by_learnability(loss_learner, loss_ref, frac):
    raise NotImplementedError

def batch_learnability(L_learner, L_ref, idx):
    raise NotImplementedError

Now run the rule on a super-batch built from embeddings, a worked example small enough to check by brute force. Ten candidate pairs each show their own concept. The reference has learned all ten. The learner confuses concepts 0 and 1, since its image embedding for either sits halfway between the two, and likewise concepts 2 and 3. It has not yet aligned the images and captions of concepts 4 and 5. Pairs 6 to 9 it already knows.

The comparison takes a sub-batch of 4 in three ways, and checks all 210 subsets of size 4 by brute force.

Greedy missed the best subset because pair 1's gain becomes large only after pair 0 is in, and pair 0 entered last. That short-sightedness is the price of part 2's heuristic. Sampling the chunks, as JEST does, softens it, because a weak-looking member of a valuable couple then has some chance of entering early. Scale softens it too, since a super-batch of hundreds of thousands of candidates holds many such members.

What online selection buys, and what it costs

Those speedups are counted in iterations, and scoring is not free. Count the arithmetic per kept example in units of $F$, the cost of one forward pass of the learner. A training step costs about $3F$, the forward pass plus a backward pass of about twice its cost (§30.2). Selection runs the learner forward on all $1/\phi$ candidates per kept example, and can reuse the kept ones' forward passes for the backward step, so the learner costs $F/\phi + 2F$. A reference with forward cost $F_{\text{ref}}$ adds $F_{\text{ref}}/\phi$, unless its outputs are cached. Relative to uniform training, the cost per kept example is

with $F_{\text{ref}} = 0$ for a cached reference. In plain words, the overhead grows like $1/\phi$, so a harder filter costs more per step, and a reference as big as the learner doubles the scoring bill.

Caching needs care. For independent selection, a reference score is one number per example and can be stored with the data. For joint selection, the scores depend on which other examples share the super-batch, which is not known in advance. So JEST stores the reference's embeddings instead and recomputes its terms $\ell_{ij}(\boldsymbol\theta_{\text{ref}})$ from them. The sigmoid loss allows this, because each term needs only its own image and text embedding and the reference's scale and bias.

Online selection fails in recognizable ways.

• The reference decides what counts as worth learning, so training is pulled toward its curated set and away from everything it never saw, the DFN lesson in another form. JEST found that, across curation strengths, a more strongly curated and weaker reference gave a better learner. For one curated set, training the reference longer helped the learner until the reference had saturated that small dataset.
• Fixed-compute benchmarks such as DataComp count training samples, so a fair comparison must also count the scoring FLOPs and the reference's training.
• Higher filtering ratios behave like larger batches, and JEST needed Adam's $\beta_2$ lowered to 0.95 to keep them stable.
• A super-batch of 163,840 candidates must be scored before each step needs it, which JEST suggests running on separate devices in parallel with training.

An interviewer who asks about online selection wants three things, why the plain highest-loss rule fails, why a contrastive batch must be chosen jointly, and what the speedup costs in FLOPs.

Synthetic captions

Filtering throws away the image along with its bad caption, and DataComp's best filters throw away 70 to 90 percent of the pool. Many of those images are fine, and only their text is bad. A captioning model, a network that writes a description of an image, can give such an image a new caption and turn a discarded pair into a useful one. Two large studies show what this buys and what it costs.

• Mix raw and synthetic captions. Nguyen et al. (2023) captioned DataComp's pools with BLIP-2, a pretrained image captioner. At the medium scale (128 million pairs), the recipe that did best on ImageNet kept raw captions for the top 30 percent by CLIP score and used BLIP-2 captions for the other 70 percent. It then dropped the synthetic captions that fell below the same CLIP-score cutoff. It reached 31.7 percent on ImageNet against 29.7 for the best raw-only filter, 0.368 against 0.328 averaged over 38 tasks, and more than doubled retrieval performance. A captioner's score on captioning benchmarks (CIDEr) did not reliably predict how useful its captions were. At 1.28 billion pairs the gain shrank. The best mix, with both sources also filtered toward ImageNet, beat the matching raw-only filter by only 1.2 points (64.3 against 63.1 percent). Synthetic captions alone (48.7 percent) fell far behind raw captions filtered by score (57.8), and filtering images became more important than fixing text.
• Recaption everything. Li et al. (2024) built a captioner in the style of LLaVA-1.5, a vision-language model, with LLaMA-3-8B as its language model, and recaptioned about 1.3 billion DataComp-1B images with long, detailed captions. They call the result Recap-DataComp-1B, and the CLIP trained on it Recap-CLIP. Their training mix uses the original caption with probability $p$ and the recaption otherwise.

Why should better captions hurt classification? Synthetic captions are better one at a time, with less noise and more visual detail, and worse as a collection. They are written by one model in one style, so their vocabulary and phrasing are narrower than the web's. Nguyen et al. measured this as fewer unique nouns and three-word phrases, with a gap that grows with scale. They describe what the captioner can see and name. So rare names, which the web's alt-text supplies and zero-shot classification asks about, become generic. The caption says "a bird on a branch" where the alt-text said "Western Kingbird".

Long captions bring a practical problem too. They do not fit the short text window of a standard CLIP text tower, 77 tokens, of which fewer really matter (§30.6), so Recap-CLIP raised its text length to 128 tokens.

Three more risks are easy to miss.

• Captioners hallucinate, so some details in a synthetic caption are false.
• A captioner trained on COCO brings COCO's style, and possibly its images, into your data, so retrieval measured on COCO is suspect. Nguyen et al. left COCO retrieval out of their large-scale comparison for this reason, since BLIP-2 had been trained on it.
• Training generation after generation on synthetic text narrows the distribution toward collapse, unless the original data is kept and accumulated (§8.4).

The common recipe answers all of these with a mix. Keep the raw text, add synthetic text, filter both, and draw one of the captions at random at each step. In an interview, the expected answer to "should we recaption everything?" is no, and the reason is variety and names, not caption quality.

import numpy as np

def mix_captions(raw_score, syn_score, policy, frac=0.3):
    raise NotImplementedError

Multimodal language models as judges

A newer kind of filter grades each pair with a multimodal language model, which reads the image and the caption together instead of embedding them separately. A fine-tuned judge of this kind filters DataComp a little better than a CLIP score. It costs about a hundred times the arithmetic per pair, so it is best run as a second stage, on pairs that cheaper filters have already kept. The box below gives the recipe, the numbers, and the failure modes.

Duplicates, contamination, and safety

Three more checks belong in every pipeline, and none of them shows up as accuracy.

Duplicates. The web repeats itself. Near-duplicate images with near-duplicate captions waste compute and make memorization more likely. Semantic deduplication, which embeds every item, clusters the embeddings, and drops near neighbors within each cluster (§8.3), removed half of a 440-million-pair LAION subset (Abbas et al. 2023). It cost less than half a point of accuracy and halved training time.

Contamination. Test images are on the web too, so a training set can contain the benchmark it is measured on. DataComp removed about 3 percent of its downloaded images as near-duplicates of an evaluation image, and ALIGN removed near-duplicates of its test sets. CLIP's own audit, discussed in §30.3, found a median overlap of 2.2 percent across 35 evaluation sets, which rarely moved accuracy by more than 0.1 points. The broader problem of measuring contamination is the subject of §22.3.

Safety and law. In December 2023 the Stanford Internet Observatory reported that LAION-5B contained links to child sexual abuse material (Thiel 2023). The material was found by combining perceptual hash matching (PhotoDNA, which fingerprints an image so that altered copies still match), cryptographic hashes, nearest-neighbor search, and classifiers. Many candidates were then verified by outside organizations. LAION withdrew the dataset at once. In August 2024 it released Re-LAION-5B, 5.5 billion pairs, after removing 2,236 links that matched hash lists from the Internet Watch Foundation, the Canadian Centre for Child Protection, and the Stanford Internet Observatory (LAION 2024). That set includes the 1,008 links that outside organizations had validated for the report. The lessons for anyone building such a set are procedural.

• Match every image against hash lists of known abuse material from the organizations that maintain them, before training and before any release, and report matches as the law requires. A CLIP-based NSFW classifier is not a substitute, since LAION-5B had one.
• Filter NSFW and toxic content (DataComp removed about 19 percent of its downloads this way), blur faces, and remove personal data, as §8.2 describes for text.
• Datasets released as URLs change after release. Images disappear or are replaced. Carlini et al. (2023) showed that this can be exploited, estimating that control of 0.01 percent of LAION-400M could have been bought for about 60 US dollars through expired domains. Publishing a cryptographic hash of every image, as DataComp does, lets every later download be verified.
• Licensing and consent for images raise the same questions as for text (§8.2), with the added issue of people photographed without their knowledge.
• Social biases in the data become biases of the model's zero-shot labels. Measuring them is part of §30.6. The general mechanism, embeddings absorbing the statistics of their corpus, is the subject of §1.9.

Choosing a curation method

The best published pipelines combine these methods. They balance concepts to set the distribution, and score image-text agreement to remove the misaligned pairs. They mix synthetic captions back into pairs whose images are worth keeping, and tune how hard to filter at the compute they will use. Where the compute allows, they choose each batch online against a curated reference. SigLIP 2 applies filters that reduce bias to WebLI's 10 billion images and selects the batches of its smallest models by learnability (§30.4). The same choices reappear in the training data of vision-language models and in the recaptioned datasets of text-to-image models.

In PyTorch

At scale, curation is batched tensor code over precomputed embeddings and match lists. The block below scores 100,000 matched pairs and keeps the top 30 percent. Then it runs metadata balancing over flat (pair, entry) index tensors without any Python loop. One observation makes balancing vectorizable. A pair's fate depends on its coins only through whether any comes up heads. So one draw per pair with probability $q_i$ has exactly the same distribution as one coin per (pair, entry).

In this toy the matches are independent of the scores, so the two filters overlap about as much as chance predicts (30 percent of 30 percent). Real pools correlate them, which is the point of combining them. The top entry keeps thousands of pairs against a cap of 20. That is the floor-not-ceiling effect of the theorem in its most extreme form, since every pair here matches three random entries. Scoring real images needs a real model, which downloads weights, so the next block is shown but not run.

Read more

• Gadre et al. (2023). DataComp: In search of the next generation of multimodal datasets. The fixed-compute benchmark, CommonPool's construction, and the baselines that produced DataComp-1B.
• Xu et al. (2024). Demystifying CLIP Data. MetaCLIP, the reconstruction of CLIP's curation, with the long-tail statistics, the balancing algorithm, and its ablations.
• Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models. CLIP-score filtering at web scale, with the thresholds, the yields, and the authors' own discussion of the filter's biases.
• Fang et al. (2023). Data Filtering Networks. Why a better model is not a better filter, and how to train one that is.
• Udandarao et al. (2024a). No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance. The log-linear law between concept frequency and zero-shot performance.
• Goyal et al. (2024). Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic. The quality-quantity trade-off and filtering that depends on compute.
• Nguyen et al. (2023). Improving Multimodal Datasets with Image Captioning. Controlled experiments on mixing raw and synthetic captions, and where synthetic text stops helping.
• Chuang et al. (2025). Meta CLIP 2: A Worldwide Scaling Recipe. Balancing extended to more than 300 languages, and the model size at which the curse of multilinguality breaks.
• Evans et al. (2024). Data curation via joint example selection further accelerates multimodal learning. JEST, learnability scored on whole batches, the chunked sampler, and the FLOP accounting of online selection.
• Mindermann et al. (2022). Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. The derivation of the reducible holdout loss that learnability scoring rests on.
• Udandarao et al. (2024b). Active Data Curation Effectively Distills Large-Scale Multimodal Models. ACID and ACED, online selection with a large reference as a form of distillation, used for SigLIP 2's small models.
• Wang et al. (2024). Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters. A fine-tuned LLaVA as a drop-in replacement for the CLIP score, with DataComp results and its cost.

Watch next

▶

▶

▶