In March 2023 a Stanford group spent less than $500 on OpenAI API calls to generate 52,000 instruction-following demonstrations, fine-tuned LLaMA 7B on them for about three hours on eight A100 GPUs, and released the result as Alpaca. In a blind comparison by the five authors on a set of test instructions, Alpaca won 90 comparisons against OpenAI's text-davinci-003 and lost 89. A chat model, apparently, for the price of a used laptop.

Two months later a Berkeley group fine-tuned a series of models on far more imitation data and looked closer. Crowd workers rated their outputs as competitive with ChatGPT. But on targeted benchmarks the imitation data did not cover, the models had closed little to none of the gap between the base model and ChatGPT (Gudibande et al. 2023). They had learned to sound like ChatGPT.

Both results are true, and between them sits everything you need to know about instruction data. Data shapes behavior quickly and cheaply, which is the Alpaca story. Data shapes capability slowly and expensively, which is the imitation story. This section covers where SFT data comes from, how to make it good and diverse, how to keep benchmarks out of it, and how to mix sources into one training set.

What one training example looks like

An SFT example is a conversation, a list of messages with roles, usually stored as one JSON object per line.

{"messages": [
  {"role": "system", "content": "You are a concise assistant."},
  {"role": "user", "content": "Convert 3.5 km to miles."},
  {"role": "assistant", "content": "3.5 km is about 2.17 miles (1 km = 0.6214 mi)."}
]}

Single-turn data is the special case of one user message and one assistant message, often written as a prompt $x$ and a response $y$. Multi-turn data has several of each. Tool-use data adds tool calls inside assistant messages and tool results as extra messages. The chat template of §2.5 turns all of it into one token sequence, and §14.3 decides which of those tokens the loss sees. Here we care only about the content.

Four sources of instruction data

Every SFT dataset in use today is built from some mixture of four kinds of source.

Human-written data is the gold standard for intent. Real users ask for things in ways no researcher would think to write. Its problem is that quality is uneven in ways you only discover after training. Llama 2's authors found that different annotation vendors produced markedly different downstream models, and that after a while their SFT model's own outputs were often competitive with what human annotators wrote, so they shifted their budget from writing demonstrations to collecting preferences (Touvron et al. 2023). That is a general pattern. Humans are much better at judging responses than at writing them, which is the premise of Chapter 16.

Templated collections such as FLAN taught the field that instruction tuning generalizes across tasks (§14.1). They still appear in modern mixes (Tulu 2 used 50,000 FLAN examples plus 50,000 chain-of-thought examples from the same collection), but their short, label-like answers teach a model to be terse, so they are diluted with conversational data.

Self-Instruct, step by step

Wang et al. (2023) asked whether a model could write its own instruction data. Their pipeline, Self-Instruct, is worth knowing in detail because nearly every synthetic data method since is a variation on it.

1. Seed. Start a task pool with 175 human-written tasks, one instruction and one example each.
2. Generate instructions. Prompt the model with 8 instructions from the pool (6 human-written, 2 previously generated) and ask for more.
3. Classify. Ask the model, few-shot, whether each new instruction is a classification task (one with a small label set).
4. Generate instances. For ordinary tasks, generate the input first and then the output. For classification tasks, generate the label first and then an input for that label (output-first), because input-first generation produced inputs biased toward one label.
5. Filter. Add a new instruction to the pool only if its ROUGE-L similarity with every instruction already there is below 0.7. Also drop instructions that mention things a text model cannot handle (images, pictures, graphs), duplicate instances, and generations that fail simple heuristics, such as an output that repeats its input.

Run on the original GPT-3, the pipeline produced about 52,000 instructions with 82,000 instances. Then the authors checked a random sample by hand. 92% of the instructions described a valid task, but only 58% of the outputs were correct, and only 54% of examples had every field valid. Fine-tuning GPT-3 on this half-wrong data still gave a 33% absolute improvement on Super-NaturalInstructions, matching InstructGPT-001. The lesson is subtle. For teaching the format of instruction following, noisy data is surprisingly fine, because what the model learns is "respond to the instruction", and that pattern is in every example, right or wrong.

The ROUGE-L filter is the part that keeps the pool from collapsing into rephrasings of the same few tasks. ROUGE-L scores two texts by their longest common subsequence of words, the longest sequence of words that appears in both in the same order, not necessarily adjacent. With $\ell$ the LCS length and $|a|$, $|b|$ the word counts, precision is $\ell/|a|$, recall is $\ell/|b|$, and the score is their harmonic mean,

In plain words, it is the fraction of words the two texts share in order, averaged over both lengths. "Write a poem about the sea." and "Write a short poem about the sea." share 6 words in order out of 6 and 7, so the score is $12/13 \approx 0.92$, a near-duplicate. Let's run the filter on 300 instructions generated from templates, a cartoon of what a model produces when sampled many times.

Of 262 distinct strings, only 64 survive. Exact deduplication would have kept all 262, but most differ by a verb or a modifier, and a model trained on them sees the same few tasks over and over. The filter is quadratic in the pool size, which is fine for 50,000 instructions with a fast LCS but not for millions, where embedding-based near-duplicate search (§7.4) or MinHash (§8.3) takes over.

import re

def rouge_l(a, b):
    raise NotImplementedError

def novelty_filter(instructions, threshold=0.7):
    raise NotImplementedError

Distillation from stronger models

The fastest way to good responses is to have a better model write them. Alpaca used Self-Instruct's recipe with text-davinci-003 as the generator. Vicuna fine-tuned on 70,000 ChatGPT conversations that users had shared publicly through ShareGPT. Its authors' headline claim, more than 90% of ChatGPT's quality, came from what their own post called a fun and non-scientific evaluation with GPT-4 as the judge, which is a fair warning about how such numbers are made. Later methods made the teacher do more work. Evol-Instruct (Xu et al. 2023) has the teacher rewrite instructions step by step into harder and more varied ones before answering them. Orca (Mukherjee et al. 2023) asked GPT-4 for explanation traces and step-by-step reasoning instead of bare answers, so the student imitates the process and not only the result. Distilled data is now the bulk of nearly every open SFT mix, and the reasoning traces of §18.5 are distillation at a much larger scale.

Two things limit it. The first is the capability ceiling from the opening. A student imitating outputs learns the teacher's style long before its knowledge, and on anything the data does not cover, it falls back on its own base model.

The second is legal, and it is not optional. Most commercial model providers' terms of use restrict using their outputs to develop competing models, and OpenAI's terms did when Alpaca was released. Alpaca's authors restricted it to non-commercial research for three stated reasons. LLaMA's license was non-commercial, the instruction data came from a model whose terms prohibit developing competing models, and the model lacked adequate safety measures. Licenses chain. A dataset generated by model $A$ carries $A$'s terms, a model trained on it inherits them, and a second dataset generated by that model may inherit them again. In practice, record the provenance of every example (which model or person wrote it, under which terms), prefer teachers whose licenses permit training on their outputs, and have the terms reviewed by someone qualified before anything ships. This book cannot give legal advice, but it can tell you that "it was on Hugging Face" is not a license.

Self-generated data without a stronger teacher

When no stronger teacher is available, or allowed, the model can bootstrap. Instruction backtranslation (Li et al. 2023) goes the other way from Self-Instruct. It starts from human-written web documents, asks a seed model to write the instruction each document would be a good answer to, has the same model score the resulting pairs, and fine-tunes on the best ones, iterating. The responses are human text, so they carry no teacher's terms or style. Magpie (Xu et al. 2024) exploits the chat template itself. Give an aligned model like Llama-3-Instruct only the template up to the start of a user message, and it writes a user query, because it has learned what users say. Sampled four million times, then answered by the same model and filtered, this produced SFT data that rivaled the official instruct model on several chat benchmarks. Both work because filtering, not generation, is where the quality comes from.

Quality over quantity, and where that stops

LIMA's ablations in §14.1 are the cleanest evidence that quality and diversity matter more than quantity for chat quality. A 16-fold increase in quality-filtered Stack Exchange data gave no improvement, while unfiltered data of the same size cost half a point. AlpaGasus (Chen et al. 2024) made the same point by subtraction. Scoring Alpaca's 52,000 examples with ChatGPT and keeping the 9,000 rated highest gave a model that significantly outperformed the original Alpaca under GPT-4 and human evaluation, and trained 5.7 times faster. Many of the discarded examples had wrong or irrelevant answers, exactly what Self-Instruct's own audit predicted.

The common quality filters, roughly in order of cost, are these.

• Rules. Length bounds, language ID, removing boilerplate refusals and "As an AI language model" disclaimers inherited from a teacher, and removing mentions of other model names. Tulu 2 filtered out every example that referred to systems such as GPT-4 or Claude, so that the model would not contradict its own hardcoded identity examples (Ivison et al. 2023).
• Deduplication, exact and near, as in §8.3, plus the ROUGE-L or embedding filters above for prompts.
• Verification, wherever correctness is checkable. Run the unit tests for code, check the final answer for math, and keep only responses that pass. This is the strongest filter there is, and it is the seed of §18.3.
• Model-based scoring, with an LLM judge (AlpaGasus) or a reward model, keeping the top fraction. Cheap and effective, but it inherits the judge's biases, notably for longer answers (§14.7).

Now the limit. For skills, more data keeps helping. Tulu 3's authors found their models improving on average as more of their SFT mix was included, with large gains on GSM8K as the math data grew, and they stopped growing it only because they wanted the remaining prompts for preference tuning (Lambert et al. 2024). The resolution is the one from §14.1. If you need a format, a few thousand excellent examples will do. If you need the model to get better at math, code, or a language, plan for hundreds of thousands, and verify them.

Diversity

A model generalizes from the prompts it saw to the prompts it will see. If the training prompts cover only "write a poem about X", it will be excellent at poems and odd at everything else. Diversity is about coverage of the space of real requests, in task type (writing, coding, math, extraction, advice, role-play, refusals), domain, difficulty, length, language, and format of the answer.

Three ways to measure it, from crude to useful. Lexical measures count distinct n-grams, or, as Self-Instruct did, extract the root verb and its direct object from each instruction ("write, poem", "summarize, article") and look at the distribution. Semantic measures embed every prompt (§7.3), cluster them, and count how many clusters are populated and how evenly. Taxonomic measures label each prompt with a category from a fixed taxonomy and compare the histogram with the traffic you expect in deployment, which is the measure that matters most, because it asks whether you cover what users will actually do.

Increasing diversity is mostly a matter of where prompts come from. LIMA sampled Stack Exchange communities with a temperature of $\tau = 3$ over sources, so that the most popular communities did not dominate, the same temperature trick that pretraining mixtures use (§8.4). Tulu 3 generated prompts conditioned on personas sampled from PersonaHub, a billion short descriptions of people, such as "a high school chemistry teacher" or "a retired sailor", so that the same request for a math problem comes out differently a hundred thousand times (Ge et al. 2024). Evol-Instruct rewrites prompts to add constraints, depth, or rarer topics. All three fight the same enemy. A model sampled many times with the same prompt produces the same few things.

Decontamination

If an evaluation question, or a light paraphrase of it, is in your SFT data, your benchmark score measures memory. Benchmarks are on the web, generated data can reproduce them from the teacher's memory, and public instruction datasets have been found to contain test sets. Decontamination for SFT works like the pretraining version in §8.3, with two differences that come from the data.

Tulu 3 describes a careful recipe (Lambert et al. 2024). It compares prompts only (the user turns), because responses in training sets are usually regenerated by models and would not match anyway. It tokenizes and matches 8-grams. A token of a test instance counts as matched if some 8-gram containing it also appears in a single training instance, and the test instance is flagged if more than 50% of its tokens are matched against the same training instance. A whole training dataset is called contaminated if it overlaps more than 2% of any evaluation's instances. Contaminated datasets were dropped, or, when that hurt too much, the matching instances were removed. Partial n-gram coverage is the point. It catches "the same math problem with different numbers", which full-string matching misses.

The verbatim copy is fully covered, and the number-changed version is still flagged at 56%, above the 50% threshold, because the long runs of words between the numbers survive intact. (Make the problem shorter and the numbers denser, and it can slip under, which is one reason to also look at lower $n$ for math.) The paraphrase gets 0% at $n = 8$. Dropping to 3-grams catches only a sliver of it (7%), at the price of false positives on any two texts that share common phrases. Yang et al. (2023) showed how much this matters. A 13B model trained on paraphrased or translated benchmark questions, which pass n-gram decontamination, reached scores on par with GPT-4 on MMLU, GSM8K, and HumanEval. They proposed checking candidate pairs with an LLM, and used it to find 8 to 18% of HumanEval overlapping popular pretraining sets. Paraphrase contamination is the open problem, and embedding search followed by an LLM check of the nearest neighbors is the practical answer today (detecting contamination after the fact is a field of its own).

import numpy as np

def matched_fraction(test_tokens, train_tokens, n=8):
    raise NotImplementedError

def contaminated_tests(test_set, train_set, n=8, frac=0.5):
    raise NotImplementedError

Mixing sources into one dataset

No single source covers everything, so every serious SFT set is a mixture. Here is Tulu 2's, a well-documented open recipe for Llama 2 (Ivison et al. 2023). The counts are the sampled sizes before the final filter for mentions of other models, which brought the total to 326,154.

Tulu 3 grew this to 939,344 examples by keeping strong general data, adding skill-specific sets for math, code, precise instruction following, safety, and multilingual chat (many generated with personas), ablating each addition, and decontaminating everything (Lambert et al. 2024). Three lessons from such mixing studies are general.

Skills compete. Adding a lot of one skill can cost another. When Tulu 2's authors fine-tuned a coding variant, its coding evaluation improved by about 70% on average while its open-ended chat scores on AlpacaEval dropped by about 20%. Every addition to a mix should be judged on the whole evaluation suite (§14.7), not on its own target.

The loss counts tokens, not examples. With the usual token-averaged loss (§14.3), a source's influence on the gradient is its share of trained tokens, which can be very different from its share of examples. A ShareGPT conversation with ten long assistant turns can carry a hundred times the tokens of a FLAN example with a one-word label.

Small, precious sets get upsampled. A few hundred hardcoded identity examples or a thousand curated LIMA examples would be drowned in a million others, so practitioners often repeat such sets several times while subsampling large ones, the same trade-off between repetition and coverage as in pretraining.

Here is how lopsided the token shares can be. The example counts are Tulu 2's. The mean trained tokens per example are invented for illustration, but they are the right shape, short for templated tasks and long for conversations.

Under these assumptions FLAN is 15% of the examples but about 1% of the trained tokens, and ShareGPT is a third of the examples and about 70% of the gradient. Whatever lengths your data really has, compute this table before you train, because it, not the example counts, is your mixture.

import numpy as np

def plan_mixture(n_examples, mean_tokens, target, budget, max_epochs=3.0):
    raise NotImplementedError

The real way

In practice the pipeline is a chain of datasets operations, run once and saved with its provenance. Here is its skeleton with the Hugging Face libraries. It is shown, not run, because it downloads data.

The real version adds near-duplicate removal, the per-instance coverage rule from Exercise 14.2b instead of the any-8-gram shortcut above, and a report of token shares per source.

Read more

• Wang et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. The bootstrapping pipeline, the ROUGE-L filter, and an honest audit of data quality.
• Taori et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model. Distillation for a few hundred dollars, and why the release was research-only.
• Ivison et al. (2023). Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. An open SFT mixture with per-source counts and ablations.
• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. Persona-driven synthetic data, decontamination details, and data scaling for skills.
• Chen et al. (2024). AlpaGasus: Training A Better Alpaca with Fewer Data. Filtering 52K examples to 9K with an LLM judge, and doing better.
• Yang et al. (2023). Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. Why n-gram decontamination misses paraphrases, and an LLM-based check.
• Xu et al. (2024). Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. Using the chat template to make a model write its own user queries.
• Li et al. (2023). Self-Alignment with Instruction Backtranslation. Writing instructions for human-written answers, then self-curating.

Watch next

▶

▶