§8.1 settled what the model is trained to do, predict the next token of real text. That objective is only as good as the text it is fed, and the model will learn whatever the corpus contains, spam and boilerplate included. So the next question is where the text comes from. GPT-2 was trained on WebText, about 40 GB of text from pages linked on Reddit. Five years later, LLaMA 3 405B read 15.6 trillion tokens, on the order of 60 terabytes of text, and the open FineWeb dataset alone holds 15 trillion. Nobody typed those tokens for us. Almost all of them come from the same place, a free archive of the public web, and almost all of the work that separates a good dataset from a bad one happens after the crawl, in deciding what to throw away.

That work is the least glamorous part of training a language model and one of the most important. The FineWeb and DCLM papers both show that, at a fixed number of tokens and a fixed architecture, the choice of filters moves benchmark scores as much as several times more compute would. This section walks through a modern web-data pipeline stage by stage, from raw HTML to clean text, with the numbers that real pipelines use and the reasons behind them.

The web, as Common Crawl sees it

Common Crawl is a nonprofit that has crawled the web regularly since 2008 and gives away the results. Each crawl is a snapshot of a few billion pages. The September 2026 crawl, CC-MAIN-2026-39, holds 2.17 billion pages, 361 TiB uncompressed, from about 40 million hosts. Almost every open and closed pretraining corpus starts from some set of these snapshots. FineWeb used 96 of them, from 2013 to April 2024 (Penedo et al. 2024).

Each crawl ships in three formats, and which one you start from matters more than you would guess.

A crawl is not the web. Common Crawl follows links from seed lists, respects robots.txt, and samples rather than exhausts most sites, so some languages, regions, and kinds of sites are underrepresented. Pages also repeat across snapshots, which matters for deduplication (§8.3). And it contains everything the web contains, including spam, machine-generated pages, adult content, and personal data.

The web is the bulk of a corpus but rarely all of it. A typical frontier mixture adds sources with different strengths, which we balance in §8.4.

From HTML to text

The first step looks trivial and isn't. A web page's HTML contains the article you want plus navigation menus, cookie banners, "related stories", comment widgets, footers, and scripts. Keeping all of it fills the corpus with millions of copies of "Accept all cookies" and "Sign in", while cutting too aggressively loses tables, code blocks, and math.

Common Crawl's own WET extraction keeps nearly all visible text, boilerplate included. FineWeb compared it with running trafilatura, a main-content extractor, on the raw WARC files, and found the trafilatura text trained clearly better models, so FineWeb extracts from WARC. DCLM ran the same comparison and measured its CORE benchmark average at 20.7 for WET text, 24.1 for the resiliparse extractor, and 24.5 for trafilatura (Li et al. 2024). They chose resiliparse anyway, because it was about eight times faster and almost as good. At the scale of hundreds of terabytes, extraction speed is a real constraint.

Extractors work on a simple observation. Main content is dense text with few links, while boilerplate is short blocks that are mostly links. Here is that heuristic in a dozen lines, a toy version of what tools like trafilatura do with many more signals.

The navigation bar and the "related" links are dropped for their link density, and the cookie banner for being short. The second paragraph keeps its sentence even though it ends in a link. Real extractors add many more signals (position in the page, tag names, text-to-markup ratio, known boilerplate patterns), but the idea is the same.

Language identification

Next we decide which language each document is in. The workhorse is the fastText language identifier (Joulin et al. 2016), a linear classifier over character $n$-gram features that recognizes 176 languages and classifies thousands of documents per second on one CPU core. Character $n$-grams are the right features because they capture spelling patterns ("sch" and "ß" for German, "ñ" and "ción" for Spanish) without needing a vocabulary or a tokenizer.

Pipelines keep a document if the classifier's probability for the target language clears a threshold. FineWeb keeps English pages with a score of at least 0.65, and Dolma uses 0.5 (Soldaini et al. 2024), where language ID alone removed 61.7 percent of the bytes. Most of the web is not English.

We can build a small version of the same idea, a naive Bayes classifier over character trigrams. For each language $\ell$, estimate the probability $p(g \mid \ell)$ of every trigram $g$ from a little training text, with add-one smoothing so unseen trigrams do not get probability zero. A document with trigrams $g_1, \dots, g_m$ is then scored by

where $Z$ normalizes over languages. In plain words, each trigram votes for the languages in which it is common, and the votes add in log space. (Naive Bayes treats the trigrams as independent given the language, which is false, so its probabilities come out far too confident. Its ranking of languages is still good.)

Two sentences of training text per language are enough to get whole sentences right. The last line shows the classic failure. "machine learning" is two English words, but it has only 16 trigrams, several of them ("chi", "ine", "ear") common in other languages too, and the classifier calls it French with probability 0.66. Real language identifiers fail the same way on short texts, code, lists of names, and pages that mix languages, which is why pipelines classify the extracted text of the whole page and why thresholds for low-resource languages need separate tuning.

import math, re
from collections import Counter

def train_langid(train, n=3):
    raise NotImplementedError

def predict_lang(model, text):
    raise NotImplementedError

Heuristic quality filters

After language ID, most English pages are still not worth training on. Some are lists of keywords stuffed for search engines, some are tables of numbers, some are a thousand lines of "Page 1 2 3 Next", some are machine-translated spam. Before anyone trained classifiers for quality, people wrote rules, and the rules still do most of the work.

The most influential set comes from DeepMind's Gopher paper (Rae et al. 2021), which built its MassiveWeb corpus with the rules below. Each one targets a specific kind of junk, which is why they are worth knowing one by one.

The stop-word rule is the cleverest. Real English prose cannot avoid function words, while keyword spam and lists avoid them completely, so requiring two of eight common ones is a nearly free test of whether the text is made of sentences.

Gopher added repetition filters, which remove a document if too much of it is repeated within itself, for example if more than 30 percent of its lines are duplicates of other lines, or if the most common 2-gram accounts for more than 20 percent of its characters (the thresholds tighten for longer $n$-grams, down to 10 percent for duplicated 10-grams). Repetitive documents are often generated, and they teach the model to loop.

C4 (Raffel et al. 2020) used a different style of rule, applied line by line. Keep only lines that end in terminal punctuation and have at least 5 words, drop pages with fewer than 3 sentences, and drop any page containing "lorem ipsum", a curly brace, or a word from a list of obscene words. Some of these rules were blunt. The curly-brace rule removes every page with code, and Dodge et al. (2021) found that the bad-words list disproportionately removed text by and about minority groups, including pages that mention LGBTQ identities and text in African American English, much of it not offensive at all.

FineWeb's ablations show how much a single rule can cost or gain (Penedo et al. 2024). C4's terminal-punctuation rule alone removed about 30 percent of the tokens, too much for what it bought, so FineWeb dropped it and kept the other C4 rules, which together remove about 7 percent. They then designed three rules of their own by comparing the statistics of high-quality and low-quality crawls. Remove a document if at most 12 percent of its lines end in punctuation (10.1 percent of tokens removed), if at least 10 percent of its characters sit in duplicated lines (12.5 percent), or if at least 67 percent of its lines are shorter than 30 characters (3.7 percent). Together these removed about 22 percent of the tokens and raised the average benchmark score of a small ablation model by about one point.

import string

def quality_flags(text):
    return []          # placeholder: keeps everything

Rules are cheap, transparent, and easy to debug, and every one of them has false positives. A poem fails the punctuation rules, a recipe fails the bullet rule, a page of mathematics fails the alphabetic-word rule. Pipelines tolerate this because the web is so large that losing some good documents costs less than keeping many bad ones, but it is also why specialized sources (code, math, poetry) are usually collected through separate pipelines with their own rules.

Here is a pipeline you can tune yourself. The corpus is synthetic, 1,500 documents of seven kinds with statistics chosen to resemble what each kind looks like on the real web, so you can see which filter removes which kind.

Model-based quality filters

Rules can say what junk looks like. They cannot say what good text looks like. For that, pipelines train a classifier to tell a small set of documents we believe are good from a random sample of the crawl, and keep the documents it scores highest.

The simplest model-based signal is perplexity. CCNet (Wenzek et al. 2020) trained a small 5-gram language model on Wikipedia for each language, scored every Common Crawl paragraph with it, and split each language's documents into head, middle, and tail thirds by perplexity, so the head is the text that reads most like Wikipedia. It is cheap, needs no labeled negatives, and works for any language with a Wikipedia, and LLaMA 1's web data came from this pipeline plus a classifier (Touvron et al. 2023). Its weakness is the one we are about to meet in every classifier, only stronger. Low perplexity under a Wikipedia model means "looks like Wikipedia", and it can also reward repetitive, predictable text.

GPT-3 used a trained classifier instead, a logistic regression on hashed word features, with WebText, Wikipedia, and books as positives and raw Common Crawl as negatives (Brown et al. 2020). Rather than keep only the top scores, it kept a document if np.random.pareto(9) > 1 - score, which keeps nearly every high-scoring document and a few low-scoring ones, to avoid narrowing the distribution too much. It filtered 45 TB of compressed crawl down to 570 GB.

DCLM ran the most thorough comparison so far (Li et al. 2024). They started from DCLM-Pool, 240 trillion tokens extracted from all of Common Crawl before 2023, applied RefinedWeb's heuristic filters, and then compared many quality filters by training models on what each one kept. The winner was surprisingly simple. A fastText classifier over word unigrams and bigrams, trained on about 400,000 documents, with positives from OpenHermes 2.5 (a set of instruction-following conversations) and highly upvoted answers from the r/ExplainLikeImFive forum, and negatives sampled from their RefinedWeb-style data. Keeping the top 10 percent gave DCLM-Baseline, 3.8 trillion tokens. A 7B model trained on 2.6 trillion of them reached 64 percent on 5-shot MMLU, close to LLaMA 3 8B's 66 percent, which was trained with 6.6 times more compute on data that was not public.

FineWeb-Edu went one step further and used a large language model as the judge (Penedo et al. 2024). LLaMA-3-70B-Instruct scored about 460,000 FineWeb pages from 0 to 5 for educational value, with a prompt describing what each score means. A small regression head on top of an embedding model (Snowflake-arctic-embed-m) was trained to predict those scores, and then run over all of FineWeb, which took about 6,000 H100 GPU-hours. Keeping pages with a predicted score of 3 or more removed 92 percent of FineWeb and left 1.3 trillion tokens (a looser threshold of 2 leaves 5.4 trillion). In their ablations, a 1.7B model trained on 350 billion tokens of FineWeb-Edu scored 37 percent on MMLU against 33 percent for FineWeb, and 57 against 46 on ARC, and it matched the final MMLU score of another dataset with almost ten times fewer tokens.

LLaMA 3 combined all of these ideas (Dubey et al. 2024), with a fastText classifier for text that resembles pages Wikipedia cites, RoBERTa-based classifiers trained on LLaMA 2's quality judgments, and separate classifiers for code and math-heavy pages.

Here is the shape of all of these, a fastText-style linear model over hashed word $n$-grams. Feature hashing maps each $n$-gram to one of $D$ buckets with a hash function (here $D$ counts buckets, not tokens), so the model needs no vocabulary and a fixed amount of memory, at the cost of occasional collisions.

Even trained on eight documents, the classifier separates explanations from spam, because the spam's vocabulary ("click", "free", "login") barely overlaps with prose. That is also its weakness. A classifier learns whatever distinguishes its positives from its negatives, and if the positives are Wikipedia, "quality" quietly becomes "sounds like Wikipedia". This is the main critique of model-based filtering, and an interview favorite.

• It narrows the distribution. Keeping the top 10 percent by one score removes whole genres (fiction, casual dialogue, local news) that a general model should still know.
• It inherits the biases of the positive set. Dialects, topics, and communities underrepresented in the positives get filtered out more often.
• It can leak benchmarks indirectly. Positives that look like exam questions (ELI5 answers, educational pages) select text that looks like benchmarks, which helps scores partly because it is closer to the test distribution. That is not contamination in the strict sense of §8.3, but it is worth remembering when comparing datasets on the same benchmarks they were tuned for.
• Gains concentrate where the classifier points. FineWeb-Edu helps knowledge and reasoning benchmarks like MMLU and ARC most. A mixture usually adds back other data so the model stays general.

import numpy as np, re, zlib

def train_quality(texts, labels, D=4096, steps=300, lr=2.0):
    raise NotImplementedError

def keep_top(model, texts, frac):
    raise NotImplementedError

Toxic content and personal data

Two more filters run in every serious pipeline, and both involve judgment calls rather than clean answers.

Toxic and adult content. The cheapest tool is a URL blocklist of adult and malicious domains, which FineWeb and RefinedWeb both apply first. Content classifiers come next. Dolma trained fastText classifiers for hate speech and NSFW text on the Jigsaw toxic-comments data and removed sentences scoring above 0.4, about 5.5 to 7.3 percent of the content (Soldaini et al. 2024). In their ablation a far stricter threshold, 0.0004, removed about 30 percent and trained slightly better models, and they still chose 0.4 so that enough tokens survived. The trade-off is real. A model that never sees toxic text is less likely to produce it, and also worse at recognizing it, which a moderation model needs to do. Longpre et al. (2023) measured exactly this pattern, so how hard to filter depends on what the model is for, and some labs prefer to filter lightly and handle behavior in post-training (Chapter 19).

Personal data. Emails, phone numbers, street addresses, and government ids appear all over the web. Pipelines detect the easy ones with regular expressions and replace them with placeholders. FineWeb anonymizes email addresses and public IP addresses and leaves other kinds of personal data, phone numbers included, in place, a common choice because phone-number patterns also match many ordinary numbers. Dolma masks emails, phone numbers, and IP addresses with tokens such as |||EMAIL_ADDRESS||| when a page has only a few of them, and drops pages that have many. Regexes catch formats, not identities (a name plus a city can be identifying), so deduplication, which removes the repeated copies that models memorize, is part of privacy protection too.

import re

def redact_pii(text):
    return text        # placeholder: redacts nothing

Licensing and consent

Whether training on copyrighted web text is legal is being argued in courts in several countries, and the answer may differ by jurisdiction and by use. We can't settle it here, but every practitioner should know the facts that shape what labs actually do.

• robots.txt is the main signal. Common Crawl obeys it, and site owners increasingly use it to block AI crawlers specifically. Longpre et al. (2024) audited the sources of C4, RefinedWeb, and Dolma and found that in a single year, from 2023 to 2024, new restrictions made about 5 percent of all C4 tokens, and 28 percent of the tokens from C4's most actively maintained, critical sources, unavailable to crawlers that respect them. Terms of service that forbid crawling covered far more.
• Licenses vary by source. Code datasets such as The Stack keep only permissively licensed repositories and let authors opt out. Wikipedia is under a share-alike license. Books and news are mostly copyrighted, which is why open datasets avoid them or use public-domain collections.
• Openly licensed corpora exist. Efforts such as the Common Pile collect only public-domain and openly licensed text, and show that competitive small models can be trained on it, though with less data than the open web offers.
• Provenance is now part of the dataset. Good datasets publish their sources, filters, and code (FineWeb, Dolma, DCLM all do), so others can audit them and so that opt-outs can be honored in later versions.

The whole pipeline, with numbers

Put together, FineWeb's pipeline runs in this order, and the order matters, since cheap filters run first and expensive ones run on what survives.

1. URL filtering. A blocklist of adult domains.
2. Text extraction. trafilatura on the raw WARC files.
3. Language ID. fastText, keep English with score at least 0.65.
4. Gopher filters. The quality and repetition rules above. After steps 1 to 4, about 36 trillion tokens remain.
5. MinHash deduplication. Within each snapshot, at a similarity of about 0.75 (§8.3). About 20 trillion tokens remain.
6. C4 rules, minus terminal punctuation. About 7 percent removed.
7. FineWeb's three custom rules. About 22 percent removed.
8. PII anonymization. Emails and public IPs replaced.

The result is 15 trillion tokens. Applying an educational classifier on top (FineWeb-Edu) leaves 1.3 trillion. For orientation, here is how the major open web datasets compare.

How do teams decide whether a filter helps? Not by reading documents, although they should do that too. The accepted method is the data ablation. Train small models (FineWeb used 1.71B parameters on 28 billion tokens) on the data with and without the filter, with everything else fixed, and compare them on a suite of benchmarks chosen to give a clear signal early in training. Small models do not predict everything about large ones, but data rankings usually transfer, and ablations are cheap enough to run by the hundred (Chapter 9 discusses when small-scale results extrapolate).

In PyTorch

The real fastText classifier is an embedding bag over hashed $n$-grams followed by a linear layer, trained with cross-entropy. Here it is in PyTorch, on the same toy data as the NumPy version.

At web scale this model runs on CPUs, millions of documents per minute across a cluster, which is the whole point. An LLM judge (FineWeb-Edu's annotator) is used only to label a few hundred thousand examples, and a cheap model learns to imitate it on everything else.

Read more

• Penedo et al. (2024). The FineWeb Datasets. The best end-to-end account of a modern web pipeline, with an ablation behind every choice, plus FineWeb-Edu.
• Li et al. (2024). DataComp-LM. A benchmark for data curation itself, with extraction, filtering, and dedup compared at several scales.
• Rae et al. (2021). Scaling Language Models, Methods, Analysis and Insights from Training Gopher. Appendix A has the MassiveWeb quality and repetition rules that most pipelines still use.
• Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Section 2.2 defines C4, and Dodge et al. (2021), Documenting Large Webtext Corpora, shows what its filters actually removed.
• Soldaini et al. (2024). Dolma. An open three-trillion-token corpus and toolkit, with careful documentation of toxicity and PII handling.
• Longpre et al. (2024). Consent in Crisis. How fast the web is closing to AI crawlers, measured source by source.
• Wenzek et al. (2020). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. The perplexity-based head, middle, and tail split, and the pipeline behind LLaMA 1's web data.
• Longpre et al. (2023). A Pretrainer's Guide to Training Data. Measures the effects of data age, quality filters, and toxicity filters on downstream behavior.

Watch next

▶

▶

▶