You have a 7B base model, 50,000 good conversations, and a node of eight GPUs. You reuse the training script from pretraining, which worked beautifully, with its peak learning rate of $3\times10^{-4}$. Two hundred steps later the training loss is lower than ever, and the model has become strange. It answers in the right format but forgets facts it knew, its code no longer runs, and its English has a faint accent of your dataset. You have just paid the full price of catastrophic forgetting, with the wrong learning rate as the down payment.

Full fine-tuning, updating every weight of the model with the SFT loss of §14.3, is the most expressive way to post-train, and the most common in frontier labs. It is also the easiest to get wrong. This section collects what actually works, from published recipes and from measurement, explains why fine-tuning learning rates are so much smaller than pretraining ones, measures forgetting on a real model and derives what reduces it, adds up the memory bill, and lists the signals that tell you a run is overfitting.

Recipes that work

Here are the SFT hyperparameters of well-documented recipes, all for full fine-tuning with AdamW.

Four regularities stand out. The learning rate is between $2\times10^{-6}$ and $2\times10^{-5}$, while the same models were pretrained with peaks of $1.5\times10^{-4}$ to $3\times10^{-4}$ (Llama 2's pretraining peak was $3\times10^{-4}$ for 7B and $1.5\times10^{-4}$ for 70B), so fine-tuning runs at roughly a tenth to a hundredth of the pretraining rate. Larger models get smaller rates. A couple of epochs is the norm for large datasets, and tiny curated sets are trained much longer, with the checkpoint chosen by evaluating generations. And batch sizes are modest, 8 to 128 sequences. Hugging Face TRL's default learning rate for SFT is $2\times10^{-5}$, which is a sensible starting point for a 7B model and probably too high for a 70B one.

Why so small? Think about how far Adam moves a weight. Adam's update for each coordinate is the bias-corrected first moment divided by the root of the second (§0.6). When the gradient keeps the same sign, $\hat{m}_t/\sqrt{\hat{v}_t} \approx \pm 1$, so each step moves a weight by about $\eta$, regardless of the gradient's size. When the gradient's sign is noisy, the steps partly cancel, and after $S$ steps the typical displacement grows more like $\eta\sqrt{S}$. A pretrained weight in a 7B model has a typical magnitude of about 0.02 (the initialization scale of §6.2, which training changes only moderately).

At the pretraining rate, a few hundred coherent steps can move a weight by more than its own size, which is a different model. At $2\times10^{-5}$, a random walk over 3,000 steps moves each weight by about 5%, enough to change behavior and small enough to keep the knowledge. This is a mental model, not a law, but it explains the table. It also suggests why larger models get smaller rates. If every weight feeding one output unit moves by about $\eta$ in a direction aligned with the input, that unit's output moves by about $\eta$ times the sum of $d$ input magnitudes, so the same per-weight step changes a wider layer's outputs more. That is the width argument behind μP, which scales Adam's hidden-layer learning rate like $1/d$. In the real runs of the widget below, 600 steps of full fine-tuning at $3\times10^{-4}$ on a 0.8M-parameter model changed each weight matrix by 6 to 13% of its norm, and on the capitals task that was enough to cut the base model's next-character accuracy on Shakespeare from 0.53 to 0.24. Small relative moves are not small changes in behavior.

import math

def sft_schedule(n_examples, batch_size, epochs, warmup_ratio, peak_lr):
    raise NotImplementedError

Catastrophic forgetting

Train a network on task A, then on task B, and its performance on A can collapse, even when a single set of weights could do both. McCloskey and Cohen named this catastrophic interference in 1989. For language models it shows up as the alignment tax of §14.1 and as the strange 7B model of the opening. Luo et al. (2023) measured it in continual instruction tuning of models from 1B to 7B parameters and found forgetting in every model, with severity increasing with scale in that range.

Why does it happen? Take the loss on the old task, $\mathcal{L}_A$, at the pretrained weights $\theta_A$, where its gradient is roughly zero. Fine-tuning moves the weights by $\Delta\theta$. A second-order Taylor expansion gives the damage,

with $\mathbf{H}_A$ the Hessian of the old loss. In plain words, forgetting is the size of the move, measured in the old task's curvature. Moving along directions the old task does not care about (small curvature) is free. Moving along directions it relies on (large curvature) is expensive, and nothing in the new task's gradient knows which is which. That one equation suggests every remedy. Move less (small learning rates, few steps, parameter-efficient methods). Move in the cheap directions (penalize the curvature-weighted distance, as EWC does). Or keep the old loss in the objective (replay).

Elastic weight consolidation (Kirkpatrick et al. 2017) takes the second idea literally. It approximates the posterior over weights after task A by a Gaussian (a Laplace approximation) with diagonal precision given by the Fisher information $F_i = \mathbb{E}_{x\sim\mathcal{D}_A,\, y \sim \pi_\theta(\cdot\mid x)}\big[(\partial \log \pi_\theta(y\mid x) / \partial\theta_i)^2\big]$, evaluated at $\theta_A$ over old-task inputs $\mathcal{D}_A$, which near a minimum matches the Hessian's diagonal and needs only first derivatives to compute. (The common empirical Fisher uses the dataset's own $y$ instead of samples from the model, which is cheaper and close to the true Fisher only when the model fits the old data well.) The new objective is

In plain words, each weight is tied to its old value by a spring whose stiffness is how much task A depends on it. Let's watch all of this in a linear model where the expansion is exact. Task A uses mostly the first 20 of 30 input directions. Task B has only 12 examples, needs a change in the last 10 directions only, and its inputs touch every direction, so plain gradient descent on B drags task A along.

Plain fine-tuning fits task B perfectly (its loss rounds to 0) and raises task A's loss from 0 to 0.78, exactly the curvature-weighted move the formula predicts, because for a quadratic loss the expansion is exact. EWC with $\lambda = 1$, using only the diagonal of the curvature as the real method does, cuts the damage 40-fold at a small cost on B, and $\lambda = 10$ cuts it more than 100-fold. (For a linear model with Gaussian noise the Fisher equals this Hessian exactly, and here it is nearly diagonal because the input directions are independent, so the diagonal loses little.) Replay of 30% old-task gradients helps too, though less in this budget. There is a weight vector that does both tasks perfectly (we built B's target from one), and EWC finds the way there that gradient descent alone does not.

At LLM scale, EWC is rarely used, because the Fisher of a pretrained model over its pretraining distribution is expensive to estimate well and the diagonal approximation is crude. The practical remedies, roughly in order of how often labs use them, are these.

1. Replay, mixing general data into the fine-tuning set. InstructGPT's PPO-ptx mixed pretraining gradients into RL (Ouyang et al. 2022). In SFT this means keeping general chat, code, and math data in every domain-specific mix, which is also why Tulu-style mixes are broad.
2. Small learning rates and short training, the table above.
3. Parameter-efficient fine-tuning. Freeze the pretrained weights and train only a small number of new ones. The standard method, LoRA, which §14.5 derives, adds to each weight matrix a trainable correction of low rank, so the model can move only in a small set of directions per matrix. Biderman et al. (2024) compared LoRA with full fine-tuning on code and math and found that LoRA learned less of the target domain but forgot less of everything else, more than weight decay or dropout achieved.
4. Weight interpolation. WiSE-FT (Wortsman et al. 2022) averages the weights of the original and fine-tuned models, $\theta = (1-\alpha)\theta_A + \alpha\theta_B$, with $\theta_B$ the fine-tuned weights and a mixing weight $\alpha \in [0, 1]$ chosen on validation data, which for CLIP improved robustness under distribution shift by several points while keeping most of the fine-tuned accuracy. Model merging generalizes this (§14.5 merges adapters).

Here are real runs of all of this on a small model, a 0.8M-parameter character GPT pretrained on Shakespeare and then fine-tuned to chat.

The runs tell a more nuanced story than "LoRA forgets less". On the format task, which sits close to what the base model does, LoRA learned more and forgot less than full fine-tuning at each of the three learning-rate settings (LoRA's rates are three times higher at each setting). On the capitals task, which requires producing letters the base model almost never writes, every configuration that learned the task without replay forgot heavily, LoRA included, because the change that teaches the task (shift the whole output distribution toward capitals) is exactly the change that damages the base task. A low-rank update can make that change easily. What protected the base model in both tasks was replay. Mixing four Shakespeare windows into each batch of twelve chats kept base accuracy within a few points of where it started, at the cost of somewhat slower learning. Forgetting is not a property of the method alone. It depends on how much the new task conflicts with the old one.

import numpy as np

def fisher_diag(per_example_grads):
    raise NotImplementedError

def ewc_penalty(theta, theta_old, fisher, lam):
    raise NotImplementedError

The memory bill

Full fine-tuning keeps the same model states as pretraining. With mixed-precision AdamW that is 2 bytes for the bfloat16 weights, 2 for the gradients, 4 for the float32 master copy, and 8 for Adam's two moments, 16 bytes per parameter before any activation (§10.1 derives it). Parameter-efficient methods, which the next two sections build, change the bill completely, because the frozen weights need no gradient, no master copy, and no optimizer state.

The LoRA and QLoRA columns count the frozen base plus rank-16 adapters on every linear layer (40M parameters for Llama 2 7B, 63M for 13B, and 207M for Llama 2 70B, at 16 bytes each), and the QLoRA column uses about 0.52 bytes per base parameter, which §14.6 derives from 4-bit weights plus their quantization constants. Activations come on top, and for long sequences they dominate. Gradient checkpointing trades a second forward pass for most of that memory, FlashAttention removes the quadratic attention term, and both are standard in fine-tuning. For full fine-tuning of a 70B model the states alone need 1.1 TB, so even fully sharded across eight 80 GB GPUs (§10.3) they do not fit, which is why such runs use 16 or more GPUs, CPU offloading, or 8-bit optimizer states.

Evaluating during fine-tuning, and overfitting signals

The InstructGPT authors reported something every fine-tuner rediscovers. Their SFT models overfit on validation loss after one epoch, yet training for more epochs kept improving both the reward model's score and human preference ratings, so they trained for 16 epochs and picked the checkpoint by reward score (Ouyang et al. 2022). LIMA's authors found that perplexity did not correlate with generation quality at all and picked checkpoints by hand (Zhou et al. 2023). Validation loss is the wrong metric to stop on.

One mechanism behind this is easy to see. Cross-entropy punishes confident mistakes without limit, so as a model memorizes its training responses it becomes overconfident on held-out tokens it gets wrong, and validation loss climbs, while the ranking of tokens, which is what greedy or low-temperature generation uses, barely changes. Here it is in a softmax classifier trained for many epochs on 60 examples.

Validation loss roughly doubles from its best value while validation accuracy hardly moves. For a chat model the equivalent of accuracy is the quality of what it generates, which is why you evaluate generations during fine-tuning, not only loss. A practical monitoring set has four parts.

• A fixed set of prompts, a few dozen covering your use cases, sampled at every checkpoint with low temperature and read by a person, or scored by a judge (§14.7).
• Cheap automatic checks, format compliance, end-of-turn emission rate, response length distribution, repetition rate, and verifiable instructions in the style of IFEval.
• Base-capability probes, a few hundred items from MMLU, GSM8K, or a code benchmark, to catch forgetting early.
• Losses, training loss per data source and validation loss on held-out data from the same mix, read as a diagnostic, not a stopping rule.

The signals that a run is overfitting or going wrong are these.

NEFTune

One cheap regularizer deserves a mention because its reported effect was so large. NEFTune (Jain et al. 2023) adds uniform noise to the input embeddings during fine-tuning only. For a sequence of length $L$ with embedding width $d$, the noise is $\boldsymbol{\epsilon}\,\alpha/\sqrt{Ld}$ with every entry of $\boldsymbol{\epsilon} \in \R^{L\times d}$ drawn from $\text{Uniform}(-1, 1)$. Each entry has variance $1/3$, so the expected squared norm of the whole noise matrix is $Ld \cdot \frac13 \cdot \frac{\alpha^2}{Ld} = \alpha^2/3$, and its norm is about $\alpha/\sqrt3$ whatever the sequence length. In plain words, the total perturbation per sequence is fixed, so longer sequences get less noise per token.

With NEFTune, fine-tuning LLaMA-2-7B on Alpaca went from 29.79% to 64.69% on AlpacaEval, with smaller gains of about 8 to 10% on other datasets. The authors attribute it to less overfitting to the formatting and wording of the instruction data, and they note that NEFTune models write longer answers, which is part of why AlpacaEval, a benchmark whose judge favors length, rewarded them (§14.7). Treat it as a regularizer worth a quick ablation, with an evaluation that controls for length. In TRL it is one argument, neftune_noise_alpha.

import numpy as np

def neftune(emb, lengths, alpha, rng):
    raise NotImplementedError

The real way

Here is a complete SFT loop in PyTorch with everything from this chapter so far. A tiny model is "pretrained" on a paragraph of text, three new special tokens are mean-initialized (§2.5), and it is then fine-tuned to spell words backward in a chat format, with masked labels, AdamW, warmup and linear decay, gradient clipping, two micro-batches normalized by their global token count, an optional NEFTune hook, and optional replay. It runs in a few seconds on a CPU.

The plain run learns the reversal task and doubles the loss on the original text. The run with a quarter of its batches drawn from the original text keeps most of it, at some cost on the new task, the same trade-off as in the widget. In a real project the loop is a configuration of an existing trainer, with the hyperparameters from the table.

Read more

• Ouyang et al. (2022). Training language models to follow instructions with human feedback. The SFT training details and the observation that validation loss overfits while quality improves.
• Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks. Elastic weight consolidation and its Laplace-approximation derivation.
• Luo et al. (2023). An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. Forgetting across domain knowledge, reasoning, and reading comprehension from 1B to 7B.
• Biderman et al. (2024). LoRA Learns Less and Forgets Less. A careful comparison of LoRA and full fine-tuning on code and math.
• Wortsman et al. (2022). Robust fine-tuning of zero-shot models. WiSE-FT, interpolating between pretrained and fine-tuned weights.
• Jain et al. (2023). NEFTune: Noisy Embeddings Improve Instruction Finetuning. The noise scaling, the AlpacaEval gains, and the analysis of overfitting and length.
• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. Hyperparameter searches for SFT at 8B and 70B.

Watch next

▶