Chapter 3 · Section 3.2 · 50 min

Neural language models

An n-gram model treats words as unrelated symbols. Suppose the training text contains "the cat ate the fish", "the dog ate the meat", "the cow ate the hay", and "the goat ate the grass" many times, and also "the sheep grazed" many times, but never once a sentence where a sheep eats. Ask a 4-gram model what follows "the sheep ate the". It has never seen that context, so it backs off to "ate the", and there more of the training eaters were meat eaters than plant eaters. Its best guess is fish or meat.

You know better, because you know that a sheep is the same kind of thing as a cow and a goat. Nothing in the counts encodes that. The 4-gram table has a separate row for "sheep ate the" and "cow ate the", and learning one row teaches the other nothing. This is the curse of dimensionality for language. A model with a free parameter for every context needs to see every context, and there are $V^{n-1}$ of them.

Bengio, Ducharme, Vincent, and Jauvin (2003) proposed the fix that every language model since has kept. Give each word a learned vector, make the prediction a smooth function of the context's vectors, and let similar words end up with similar vectors because that helps prediction. Then whatever the model learns about cows transfers to sheep for free. This section builds their model, derives its gradients, trains it, and shows exactly where it beats counting.

The model is an embedding, a hidden layer, and a softmax

Fix a context size of $n-1$ previous tokens, exactly as in an n-gram model. The neural probabilistic language model computes the next-token distribution in four steps.

Look up. Each context token $w_{t-j}$ indexes a row of an embedding matrix $\mathbf E \in \R^{V \times d}$ , the same lookup table as §1.2.
Concatenate. Stack the $n-1$ vectors into one long vector $\mathbf x = [\mathbf E_{w_{t-n+1}}; \dots; \mathbf E_{w_{t-1}}] \in \R^{(n-1)d}$ . The order matters, so the model knows which word was immediately before and which was further back.
Hidden layer. Compute $\mathbf a = \tanh(\mathbf W_1 \mathbf x + \mathbf b_1) \in \R^{d_h}$ , with $\mathbf W_1 \in \R^{d_h \times (n-1)d}$ and $\mathbf b_1 \in \R^{d_h}$ .
Output. Compute one score per vocabulary word and normalize,

\mathbf y = \mathbf b_2 + \mathbf W_3\, \mathbf x + \mathbf W_2\, \mathbf a \;\in \R^{V}, \qquad q(w_t = i \mid w_{t-n+1:t-1}) = \operatorname{softmax}(\mathbf y)_i ,

with $\mathbf W_2 \in \R^{V \times d_h}$ , $\mathbf b_2 \in \R^V$ , and an optional $\mathbf W_3 \in \R^{V \times (n-1)d}$ .

In plain words, the context words are turned into vectors, a small neural network mixes them, and a softmax turns the network's scores into a probability for every possible next word. The term $\mathbf W_3 \mathbf x$ is the paper's direct connections from the embeddings straight to the output, a linear shortcut that is optional (set $\mathbf W_3 = \mathbf 0$ to remove it). In the paper's own symbols the output is $\mathbf b + \mathbf W\mathbf x + \mathbf U\tanh(\mathbf d + \mathbf H\mathbf x)$ , so their $\mathbf H, \mathbf d, \mathbf U, \mathbf b, \mathbf W$ are our $\mathbf W_1, \mathbf b_1, \mathbf W_2, \mathbf b_2, \mathbf W_3$ .

Everything is trained jointly, the embeddings included, by maximizing the log-likelihood of the training text, which is the same as minimizing the cross-entropy loss that §3.1 called the log of perplexity,

\mathcal L(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log q(w_t \mid w_{t-n+1:t-1};\, \theta), \qquad \theta = \{\mathbf E, \mathbf W_1, \mathbf b_1, \mathbf W_2, \mathbf b_2, \mathbf W_3\}.

The paper also added a weight decay penalty on the weights and the embedding table (not the biases), which you will see matters a great deal on small data.

Counting the parameters shows where the cost is

The parameter count is

\underbrace{Vd}_{\mathbf E} + \underbrace{(n-1)d\,d_h + d_h}_{\mathbf W_1,\ \mathbf b_1} + \underbrace{V d_h + V}_{\mathbf W_2,\ \mathbf b_2} + \underbrace{V(n-1)d}_{\mathbf W_3\ (\text{optional})} .

Take the paper's best Brown configuration, $V = 16{,}383$ , a context of $n - 1 = 4$ words, $d = 30$ , $d_h = 100$ , and no direct connections. The embedding table has $491{,}490$ parameters, the hidden layer $12{,}100$ , and the output layer $1{,}654{,}683$ , for $2{,}158{,}273$ in total. More than three quarters of the model is the output layer, and the expensive part of every forward pass is the same, the $V \times d_h$ matrix multiply that produces a score for every word. A 5-gram table, by contrast, has $V^5 \approx 1.2 \times 10^{21}$ possible entries. The neural model has about two million parameters and assigns a sensible probability to every one of those 5-grams.

Warm-up

Count the parameters of a neural language model

Implement nplm_params(V, d, n, dh, direct) returning the number of parameters of the model above (embedding table, hidden layer with bias, output layer with bias, and the direct connections if direct is true).

def nplm_params(V, d, n, dh, direct):
    return V * d + n * d * dh + V * dh

Hidden tests, hints, and AI help, from $1.99 a month.

Why it generalizes better than counts

The answer is smoothness. The predicted distribution is a continuous function of the context vector $\mathbf x$ . If two words have nearby embeddings, swapping one for the other in the context moves $\mathbf x$ only a little, so the prediction moves only a little. And training makes similar words nearby, because whenever cow and sheep appear in similar contexts, the gradient pushes their embeddings in similar directions to predict the same next words. Statistical strength flows between every context that shares a word, which is exactly what the n-gram table lacks.

Let's test the opening example. The training text below has three meat eaters and three plant eaters. Every animal appears in sentences like "the cow grazed" or "the fox hunted", but the sheep is never seen eating. We train a context-3 neural model in NumPy and compare it with a 4-gram model that backs off to the longest context it has seen.

Python · cell 1

The training loop is the forward pass and the backward pass we derive in the next part, written out in a few lines. Read it now for the shape of the computation, and come back to it after the derivation.

Python · cell 2

For the familiar context "cow ate the", both models agree, grass or hay. For "sheep ate the", the n-gram model has to back off to "ate the" and guesses fish or meat about 60% of the time, because that is what most training eaters ate. The neural model puts almost all its probability on grass and hay. It never saw a sheep eat. It saw sheep behave like cows and goats in other sentences ("grazed"), which placed the sheep's embedding where cow-like predictions come out. That is generalization through shared representations, in the smallest example that shows it.

The gradients, layer by layer

Training needs $\partial \mathcal L / \partial \theta$ for every parameter. The derivation is the backpropagation recipe of §0.3 applied to four operations, and it is worth doing once by hand because every language model in this book has the same last two steps.

For one example with target $w$ (so the loss is $\ell = -\log \operatorname{softmax}(\mathbf y)_w$ ), write $\mathbf p = \operatorname{softmax}(\mathbf y)$ . Going backward,

\frac{\partial \ell}{\partial \mathbf y} = \mathbf p - \mathbf e_w, \qquad \frac{\partial \ell}{\partial \mathbf W_2} = (\mathbf p - \mathbf e_w)\,\mathbf a^\top, \qquad \frac{\partial \ell}{\partial \mathbf a} = \mathbf W_2^\top(\mathbf p - \mathbf e_w),

\boldsymbol\delta = \frac{\partial \ell}{\partial \mathbf a} \odot (1 - \mathbf a \odot \mathbf a), \qquad \frac{\partial \ell}{\partial \mathbf W_1} = \boldsymbol\delta\,\mathbf x^\top, \qquad \frac{\partial \ell}{\partial \mathbf x} = \mathbf W_1^\top \boldsymbol\delta + \mathbf W_3^\top(\mathbf p - \mathbf e_w),

with $\partial\ell/\partial\mathbf b_2 = \mathbf p - \mathbf e_w$ , $\partial\ell/\partial\mathbf b_1 = \boldsymbol\delta$ , and $\partial\ell/\partial\mathbf W_3 = (\mathbf p - \mathbf e_w)\mathbf x^\top$ .

In plain words, the error at the output is "predicted probabilities minus the one-hot target", every weight matrix gets the outer product of the error arriving at its output with the activation arriving at its input, and the error passes backward through the tanh by multiplying with its derivative $1 - \tanh^2$ .

∂ Each step, with shapes

Softmax plus cross-entropy. With $\ell = -y_w + \log\sum_i e^{y_i}$ , the derivative with respect to $y_i$ is $-\mathbb 1[i = w] + e^{y_i}/\sum_j e^{y_j} = p_i - \mathbb 1[i = w]$ . So $\partial\ell/\partial\mathbf y = \mathbf p - \mathbf e_w \in \R^V$ . This is the result §0.2 derives in general.

Output layer. $y_i = b_{2,i} + \sum_k W_{2,ik} a_k + \dots$ , so $\partial\ell/\partial W_{2,ik} = (\partial\ell/\partial y_i)\, a_k$ , which is the outer product $(\mathbf p - \mathbf e_w)\mathbf a^\top \in \R^{V \times d_h}$ . And $\partial\ell/\partial a_k = \sum_i (\partial\ell/\partial y_i) W_{2,ik}$ , which is $\mathbf W_2^\top(\mathbf p - \mathbf e_w) \in \R^{d_h}$ .

Through the tanh. $a_k = \tanh(u_k)$ with $\mathbf u = \mathbf W_1\mathbf x + \mathbf b_1$ , and $\tanh'(u) = 1 - \tanh^2(u)$ , so $\boldsymbol\delta = \partial\ell/\partial\mathbf u = (\partial\ell/\partial\mathbf a) \odot (1 - \mathbf a^2)$ .

Hidden layer. The same outer-product rule gives $\partial\ell/\partial\mathbf W_1 = \boldsymbol\delta\mathbf x^\top \in \R^{d_h \times (n-1)d}$ . The input $\mathbf x$ feeds both $\mathbf W_1$ and $\mathbf W_3$ , so its gradient is the sum of both paths.

Embeddings. $\mathbf x$ is a concatenation, so split $\partial\ell/\partial\mathbf x$ into $n-1$ blocks of size $d$ . Block $j$ is the gradient for the embedding row of the $j$ -th context word, and it is added to that row. When the same word appears twice in a context, or many times in a minibatch, the contributions accumulate, which is the scatter-add of §1.2. For a minibatch, average the per-example gradients (or equivalently divide $\mathbf p - \mathbf e_w$ by $B$ before multiplying, as the cell above does).

In code we store each weight matrix transposed, as W1 with shape ((n-1)d, dh), so that a minibatch of row vectors can be multiplied on the left (x @ W1). The math and the code compute the same thing.

Interview

Forward pass and loss of an MLP language model

Implement nplm_loss(P, X, Y). P is a dict with E (V, d), W1 ((n-1)d, dh), b1 (dh,), W2 (dh, V), b2 (V,), and W3 either None or ((n-1)d, V). X holds context ids with shape (B, n-1), oldest word first, and Y holds the (B,) target ids. Return (loss, logits), where the loss is the mean cross-entropy and the logits have shape (B, V). Concatenate the context embeddings in order, oldest first, and compute the loss stably.

import numpy as np

def nplm_loss(P, X, Y):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Core

Backpropagate through the MLP language model

Implement nplm_grads(P, X, Y) returning a dict of gradients of the mean loss with respect to E, W1, b1, W2, b2 (no direct connections in this exercise, P["W3"] is None). The tests compare with finite differences and include a batch where the same word appears many times.

import numpy as np

def nplm_grads(P, X, Y):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

A makemore-style character model

Words are not the only choice of token. Andrej Karpathy's makemore series rebuilds the Bengio model at the character level, which is a perfect laboratory because the vocabulary is tiny and you can watch everything. Below, the same model trains live in your browser on the 435-word story from §3.1, predicting each character from the previous few. The dashed line is the best held-out perplexity of any add-k n-gram model, tuned over both $n$ and $k$ .

Three things are worth noticing. First, with the default weight decay of 0.003 the neural model ends below the tuned n-gram baseline on held-out text (in our runs it settles around 4.5 to 4.6, against 4.87 for the best add-k model, a 4-gram with $k = 0.03$ ), even on this tiny corpus. Second, with weight decay set to zero, training perplexity keeps falling while held-out perplexity bottoms out near 5.1 after about a thousand steps and then climbs past 10. The model is memorizing the story, and weight decay (the paper's choice too) is what keeps it honest. Third, look at the embedding panel. With no instruction, the vowels tend to end up apart from most consonants, because they behave alike as predictors (each is likely after a consonant and rarely after another vowel). On a story this small the effect is modest and the layout changes with the settings, but a grouping by behavior is something counting has no way to express.

⚠️ Check the loss at initialization

Before training, a model that knows nothing should be uniform over the vocabulary, so its loss should be $\log V$ . For the 26 characters of the widget's story that is $\log 26 \approx 3.26$ (perplexity 26), and for the 16 tokens of the animal corpus it is $\log 16 \approx 2.77$ , which is what the PyTorch block below prints at initialization. If the first loss you see is much larger, the output layer is initialized too large and the model starts out confidently wrong. Training then wastes its first steps undoing that confidence, and the hidden layer can saturate on the way. Initialize the output weights small, as the animal-corpus cell does with a standard deviation of 0.01. This one-line sanity check, emphasized in Karpathy's makemore lectures, catches a surprising number of bugs. §0.7 covers initialization in depth.

In PyTorch

The production version is a few lines. nn.Embedding does the lookup, view(B, -1) concatenates, and F.cross_entropy fuses the log-softmax and the negative log-likelihood into one numerically stable operation. Here it is trained on the same animal corpus as the NumPy cells.

🔥 In PyTorchfor a machine with PyTorch

At the scale of real models, two details change. The output layer's weights are often tied to the embedding table, using $\mathbf E$ as $\mathbf W_2$ (with $d_h = d$ ), which saves $Vd$ parameters and often helps (Press and Wolf 2017). GPT-2 does this, as you will see in §6.2. And the hidden layer became much wider and deeper, but its job of mixing context vectors and passing them to a softmax is unchanged. The feed-forward block inside every Transformer (§5.3) is a descendant of this hidden layer, although it acts on one position at a time and leaves the mixing of context to attention.

🔬 The softmax is a bottleneck in two ways

Computationally, the $V \times d_h$ output multiply dominates the cost, which motivated hierarchical softmax (Morin and Bengio 2005), importance sampling, and the negative sampling of §1.5. Today's accelerators just compute the full softmax for vocabularies of 50,000 to 250,000 tokens. Expressively, without direct connections every logit vector has the form $\mathbf W_2\mathbf a + \mathbf b_2$ , and a log-probability vector differs from its logits only by subtracting the same normalizer from every entry. So if you stack the log-probability vectors of all contexts into a matrix, its rank is at most $d_h + 2$ , however many contexts and words there are. Yang et al. (2018) argued that natural language may need higher rank than that and called this the softmax bottleneck, proposing a mixture of softmaxes as a fix. Large hidden sizes have made it less pressing in practice.

What the MLP model still cannot do

The Bengio model fixed the generalization problem of counts, but it kept the n-gram's other limitation. Its context is a fixed window of $n-1$ tokens, and every position in the window has its own block of $\mathbf W_1$ . That has two costs. Widening the window grows $\mathbf W_1$ linearly, and the model must learn separately what cat means as the last word, as the second-to-last, and so on, because there is no sharing across positions. Nothing beyond the window can ever influence a prediction, no matter how important.

	n-gram (Kneser-Ney)	MLP language model
parameters	one per seen n-gram, grows with data	fixed, about $Vd + (n-1)d\,d_h + Vd_h$
unseen contexts	backs off to shorter contexts	smooth function of embeddings, so similar words share
training	counting, minutes on a CPU	gradient descent, the softmax over $V$ dominates
context length	fixed $n-1$ , sparsity limits $n$ to about 5	fixed $n-1$ , parameters grow linearly with it
what it learns about words	nothing	embeddings, as a byproduct

What we want is a model that reads a context of any length, shares the same weights at every position, and summarizes everything it has read into a vector. The obvious way to do that is to apply one function over and over, feeding its own output back in. That is a recurrent network, and it is §3.3.

🎯 Interview questions on this section

Why does a neural language model generalize better than an n-gram model?

An n-gram model has an independent parameter per context, so nothing learned about "cow ate the" transfers to "sheep ate the". The neural model maps context words to embeddings and computes the prediction as a smooth function of those vectors. Words that appear in similar contexts get similar embeddings during training, so contexts that differ by a similar word get similar predictions. This shares statistical strength across the exponentially many contexts and assigns sensible probabilities to combinations never seen in training.

Where do most of the parameters and most of the compute of the Bengio model go?

To the output layer. Its weight matrix is $V \times d_h$ , which for $V = 16{,}383$ and $d_h = 100$ is about 1.6 million of the model's 2.2 million parameters, and every forward pass must compute all $V$ logits to normalize the softmax. This is why so much early work (hierarchical softmax, importance sampling, negative sampling, NCE) went into avoiding the full softmax, and why tying the output matrix to the embedding table saves so much.

What loss should you see at initialization for a language model with vocabulary size V, and why does it matter?

About $\log V$ , the cross-entropy of a uniform prediction. A much larger initial loss means the logits start large and the model is confidently wrong, which wastes early training and can saturate activations. It is fixed by initializing the output layer small (or to zero). Checking the initial loss against $\log V$ is a fast sanity test for bugs in the loss, the targets, or the initialization.

How is word2vec related to the Bengio neural language model?

Word2vec keeps the idea of learning embeddings by prediction but removes the expensive parts. CBOW averages the context embeddings instead of concatenating them and drops the hidden layer, and skip-gram predicts context words from a single center word. Both replace the full softmax with hierarchical softmax or negative sampling. The simplification made it possible to train on billions of words, which mattered more for embedding quality than the model's expressiveness.

Why did mixing the neural model with an interpolated trigram reduce perplexity further?

The two models make different errors. The n-gram model is very good at memorized, frequent phrases and exact rare collocations, and the neural model is good at smoothing across similar words. Because the log is concave, the log loss of an averaged prediction is never worse than the average of the two models' log losses, and when their errors are partly independent the mixture often beats both, the same reason ensembles help in general. Bengio et al. found the mixture always helped.

What are the main limitations of a fixed-window MLP language model?

The context is limited to $n-1$ tokens, so any dependency longer than the window is invisible. Each window position has its own weights, so the parameter count grows linearly with the window, and nothing learned about a word at one position transfers to another position. Recurrent networks address both by applying the same weights at every step and carrying a summary state of unbounded history.

🧭 What to remember

Bengio et al. (2003) replaced the n-gram table with embeddings, a concatenation of the context vectors, one tanh hidden layer, and a softmax over the vocabulary, all trained jointly by cross-entropy.
The prediction is a smooth function of the context embeddings, and training makes similar words similar, so the model generalizes to word combinations it never saw.
Backpropagation gives $\mathbf p - \mathbf e_w$ at the output, outer products for the weights, $1 - \tanh^2$ through the hidden layer, and a scatter-add into the embedding table.
The output layer dominates both the parameters and the compute, which is why the softmax drew so much engineering attention.
On small data, weight decay is essential, and checking that the initial loss is about $\log V$ catches bugs early.
The model still has a fixed window with separate weights per position. Sharing one function across positions and carrying a running state leads to recurrent networks in §3.3.

Bengio, Ducharme, Vincent, and Jauvin (2003). A Neural Probabilistic Language Model. The original paper, readable today. Section 1 on the curse of dimensionality is the clearest statement of why embeddings help.
Karpathy (2022). Building makemore Part 2, MLP. Rebuilds this exact model on character-level names, with every implementation detail and the initialization pitfalls.
Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space. Starts from the neural language model and strips it down to CBOW and skip-gram.
Press and Wolf (2017). Using the Output Embedding to Improve Language Models. Weight tying between the input and output tables.
Yang, Dai, Salakhutdinov, and Cohen (2018). Breaking the Softmax Bottleneck. The rank argument for the output layer.

Zero to Frontier is made for larger screens

Neural language models

The model is an embedding, a hidden layer, and a softmax

Counting the parameters shows where the cost is

Count the parameters of a neural language model

Why it generalizes better than counts

The gradients, layer by layer

Forward pass and loss of an MLP language model

Backpropagate through the MLP language model

A makemore-style character model

In PyTorch

What the MLP model still cannot do

Read more

Watch next