Look at what the last two sections built. The MLP language model of §3.2 learned a static embedding for every word as a byproduct of predicting the next one. The RNN of §3.3 does something more. At every position it computes a hidden state $\mathbf h_t$ that depends on the word and on everything before it. Read "she deposited the check at the bank" and the state at bank has seen deposited and check. Read "the canoe drifted toward the bank" and the state at bank has seen canoe and drifted. Same word, two different vectors.

That is exactly what §1.9 asked for. A contextual word vector is a representation of a word computed from the whole sentence it appears in, so that bank gets a different vector in every sentence. This section is about how to get good ones, and the answer that changed NLP in 2018 was surprisingly direct. Train a large language model on unlabeled text, then use its hidden states as word vectors. The system that made this work at scale was ELMo (Peters et al. 2018), Embeddings from Language Models, and its ideas (bidirectional context, using all layers, freezing versus fine-tuning, and measuring how contextual a vector is) are the backdrop for BERT and GPT.

The same word gets a different vector in each sentence

Formally, a static embedding is a function of the word alone, $\mathbf v(w)$. A contextual embedding is a function of the word and its sentence $s = (w_1, \dots, w_T)$, $\mathbf r(w_t; s) = f_\theta(s)_t$. The static case is the special case where $f$ ignores everything but $w_t$.

Before any equations, play with a toy. The widget below has eight sentences that use bank, four about money and four about rivers. It is a hand-built illustration, not a trained model. Each word starts with a hand-placed 2-D static vector. A two-layer bidirectional "contextualizer" then adds to each token's vector a decaying average of the content words on its left and another of the content words on its right (a stand-in for a bidirectional RNN, with no learned weights), and a weighted sum combines the layers, as ELMo does.

At layer 0 all eight vectors sit on the static point, so self-similarity is 1 and the sense read-off scores 50%, no better than a coin (the static bank leans financial, so all eight get called financial). Each contextual layer pulls the eight vectors toward their neighbors, the money sentences one way and the river sentences the other, and self-similarity falls. Lengthen the context reach $\lambda$ and the vectors see more of the sentence. Raise the context strength $\beta$ far enough and each vector is dominated by its neighbors rather than by the word itself, which is contextual but no longer about the word. Good contextual vectors need both the word and its context.

Language model hidden states are contextual vectors

An RNN language model reads left to right. Its state at position $t$, which we now write $\overrightarrow{\mathbf h}_t$, summarizes $w_1, \dots, w_t$, and it is trained to predict $w_{t+1}$. For "the bank raised its interest rates", that state at bank has seen only "the bank", which says nothing about the sense. The disambiguating words are on the right.

So run a second language model backward, from the end of the sentence to the start. Its state $\overleftarrow{\mathbf h}_t$ summarizes $w_t, \dots, w_T$, and it is trained to predict $w_{t-1}$. The two states together cover the whole sentence, and concatenating them, $[\overrightarrow{\mathbf h}_t; \overleftarrow{\mathbf h}_t]$, gives a vector for $w_t$ that has seen every word.

Why two separate language models instead of one bidirectional network trained to predict the next word? Because a bidirectional state cheats. The backward half of the state at position $t$ has already read $w_{t+1}$, so predicting $w_{t+1}$ from it is copying, not modeling. The cell below proves it on random tokens, where there is nothing to learn and the best honest accuracy is chance (1 in 8).

The forward state scores at chance, as it must. The bidirectional state "predicts" the next random token most of the time, because it contains it. A model trained that way would learn to read its own answer and nothing about language. This leakage problem is fundamental, and it is the reason ELMo trains two directional models, and the reason BERT (§7.1) instead hides the target word behind a mask.

The biLM objective

ELMo's bidirectional language model (biLM) is a forward LSTM language model and a backward LSTM language model trained together. They share the token representation (parameters $\Theta_x$) and the softmax layer ($\Theta_s$), and each direction has its own LSTM parameters. The objective maximizes the sum of both log-likelihoods,

In plain words, each direction is an ordinary language model, and they are tied at the input and the output so that they learn a common vocabulary of vectors. Neither direction ever sees the word it is predicting.

import numpy as np

def bilm_loss(Hf, Hb, W, b, ids):
    z = np.concatenate([Hf, Hb]) @ W + b             # every state predicts its own position: leakage
    y = np.concatenate([ids, ids])
    z = z - z.max(1, keepdims=True)
    return float(np.mean(np.log(np.exp(z).sum(1)) - z[np.arange(len(y)), y]))

The two directions see different halves of the sentence

Here is the payoff, measured. The PyTorch block below trains a small forward and a small backward LSTM language model on sentences where the sense of bank is signaled on one side. In training, cues appear on the left, on the right, or both. At test time each sentence has its cue on only one side. A nearest-centroid probe then tries to read the sense off the vector for bank.

The pattern is exactly what the theory predicts. When the cue is on the left, the forward state reads the sense perfectly and the backward state is at chance. When the cue is on the right, it is the other way around. The concatenation works in both cases, though not always perfectly, since half of the concatenated vector is uninformative for these sentences and the probe is a crude nearest-centroid rule. A static vector is at 50% by construction, since it is the same vector every time. Notice also why the forward model kept the sense in its state at all. In training, the words after bank depend on the sense, so remembering it lowered the forward model's loss. Language models keep what helps predict, and that turns out to be a great deal of meaning.

In practice the two directions are run as one bidirectional RNN layer over a padded batch, and that is where the classic bug lives. Sentences have different lengths, so a batch is padded on the right. The forward direction can run straight through and ignore its outputs past each sentence's end. The backward direction cannot. It must start at each sentence's own last real word, because if it starts at the end of the padded array, every backward state has read padding first. PyTorch's pack_padded_sequence exists to get this right, and the bidirectional encoder of §3.6 is exactly this layer.

import numpy as np

def birnn(X, lengths, Pf, Pb):
    raise NotImplementedError

ELMo uses every layer of a deep biLM

Peters et al. (2018) scaled this recipe up and added one more idea. Their biLM has a character-level input (a CNN over characters with 2048 n-gram filters, two highway layers, and a projection to 512 dimensions, so any string gets a token vector) followed by $L = 2$ layers of bidirectional LSTMs with 4096 units and 512-dimensional projections, with a residual connection from the first layer to the second. It was trained for 10 epochs on the One Billion Word Benchmark and reached an average forward and backward perplexity of 39.7.

For each token $k$ this gives $L + 1 = 3$ representations, the context-free token layer $\mathbf h^{\text{LM}}_{k,0}$ and the two biLSTM layers $\mathbf h^{\text{LM}}_{k,j} = [\overrightarrow{\mathbf h}_{k,j}; \overleftarrow{\mathbf h}_{k,j}]$ for $j = 1, 2$. Earlier work would just take the top layer. ELMo lets each downstream task learn its own weighted mix,

where the $L+1$ raw weights $\mathbf a^{\text{task}}$ and the scalar $\gamma^{\text{task}}$ are learned with the task, and the biLM itself stays frozen. In plain words, each task chooses how much it wants from the context-free layer, the lower layer, and the upper layer, and $\gamma$ rescales the result to whatever size the task model likes. The softmax keeps the weights positive and summing to one. The ELMo vector is then concatenated with the task's own word embedding at the input of the task model, $[\mathbf x_k; \operatorname{ELMo}_k^{\text{task}}]$, and the paper also found it useful to add it at the task model's output.

import numpy as np

def scalar_mix(H, a, gamma):
    return gamma * np.tensordot(a, H, axes=1)          # forgot the softmax

What each layer knows

Using all layers mattered because different layers encode different things. Peters et al. probed each layer with simple supervised tasks. For part-of-speech tagging, a syntactic task, the first biLSTM layer gave higher accuracy than the top layer. For word sense disambiguation, a semantic task, the top layer was better (an F1 of 69.0 with a nearest-neighbor classifier on the top layer, competitive with dedicated systems). Lower layers lean toward syntax and local form, upper layers toward meaning in context. Every task wants a different mix, which is exactly what the learned weights $\mathbf s^{\text{task}}$ provide. The same layer-wise picture was later found in Transformers, and probing it is the subject of linear probes in interpretability research.

The results were a jolt. Adding ELMo to existing models improved the state of the art on all six benchmarks the paper tried, question answering (SQuAD), textual entailment (SNLI), semantic role labeling, coreference resolution, named entity recognition, and sentiment (SST-5), with relative error reductions from about 6% (SNLI) to about 25% (SQuAD). One pretrained model, frozen, helped every task. That is the template the whole field has followed since.

Measuring how contextual a vector is

If contextual vectors differ across sentences, by how much? Ethayarajh (2019) defined three measurements that are now standard. Take a word $w$ that occurs in $n$ different sentences, and let $\mathbf r_1, \dots, \mathbf r_n$ be its vectors from one layer.

• Self-similarity is the average cosine between the word's vectors in different contexts, $\operatorname{SelfSim}(w) = \frac{1}{n(n-1)}\sum_{i \ne j}\cos(\mathbf r_i, \mathbf r_j)$. A static embedding has self-similarity 1. Lower means more context-specific.
• Intra-sentence similarity is the average cosine between each word's vector in a sentence and the sentence's mean vector. High means the layer makes all the words of a sentence alike.
• Maximum explainable variance is the fraction of the variance of $\mathbf r_1, \dots, \mathbf r_n$ explained by their first principal component, $\sigma_1^2 / \sum_i \sigma_i^2$. It is the best a single static vector could do at explaining the word's contextual vectors.

There is a trap. §1.9 showed that embedding spaces are anisotropic, with all vectors in a narrow cone, so even unrelated words have a high average cosine. A self-similarity of 0.6 means little if two random words already score 0.5. Ethayarajh therefore subtracts a baseline, the average cosine between the vectors of randomly sampled words in the same layer,

His findings, for ELMo, BERT, and GPT-2, were that upper layers produce more context-specific representations than lower ones, that representations are highly anisotropic (especially in GPT-2's upper layers), and that "on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word". Contextual vectors are not small perturbations of static ones. They genuinely move.

import numpy as np

def self_similarity(R):
    Rn = R / np.linalg.norm(R, axis=1, keepdims=True)
    return float((Rn @ Rn.T).mean())                  # includes the diagonal of ones

def adjusted_self_similarity(R, Rand):
    return self_similarity(R) - self_similarity(Rand)

The cell below applies these measurements to a toy, a fixed bidirectional contextualizer (two layers of the decaying left and right summaries from the widget) on random 16-dimensional word vectors. It is not a trained model, but it shows the measurement pipeline and the expected trend.

Layer 0 is static, so the raw self-similarity is exactly 1. The upper layers lower it, and the anisotropic offset keeps every raw number high, which is why the baseline matters. Notice that the baseline itself rises with depth, because mixing in neighbors makes all the words of a sentence more alike. Read the adjusted column, not the raw one. It falls from about 0.27 at the static layer to about 0.04 at layer 2, so in the upper layer almost all of the raw 0.98 is the shared offset, and what is left is context.

Feature extraction or fine-tuning

ELMo is a feature extractor. The biLM is trained once, frozen, and its layers are mixed and fed to a separate task model. A competing idea arrived the same year. ULMFiT (Howard and Ruder 2018) pretrained an LSTM language model and then fine-tuned the whole network on the target task, with tricks to avoid destroying what pretraining had learned (discriminative learning rates per layer, a slanted triangular schedule, gradually unfreezing layers from the top). On six text classification datasets it cut error by 18 to 24% on most, and with only 100 labeled examples it matched training from scratch on 100 times more data. CoVe (McCann et al. 2017) had earlier shown that the encoder of a translation model also provides useful contextual vectors, but translation data is scarce and language modeling data is not.

Which is better? Peters, Ruder, and Smith (2019) compared the two directly and found that it depends on how similar the pretraining task is to the target task. Fine-tuning usually wins when they are close, and freezing can be competitive or better when they are far apart. The trade-offs are practical as much as statistical.

Both lines of work continue today. Fine-tuning became the default with GPT and BERT, and parameter-efficient variants like LoRA (§14.5) make it cheap enough to keep one base model and many small adapters. Frozen encoders never went away either, since a retrieval system that embeds a billion documents (§7.3) cannot re-encode them every time a task changes.

What contextual vectors still lack

ELMo's vectors are contextual, but they are built by two LSTMs that each read the sentence one word at a time, compressing everything they have read into a fixed-size state. The state at bank in the backward model is a summary of the entire right context, squeezed through a few thousand numbers and many steps of recurrence. The two directions never interact inside the biLM, they are only concatenated at the end. And each layer takes $O(T)$ sequential steps to compute.

These are the same limits as in §3.3, and they matter most when a whole input sentence must be squeezed into one vector and then expanded into another sentence. That is the problem of translation, and it is where the bottleneck becomes impossible to ignore (§3.5).

Read more

• Peters et al. (2018). Deep Contextualized Word Representations. ELMo. Sections 3 and 5 (the biLM, the scalar mix, and what the layers capture) are the essential parts.
• Ethayarajh (2019). How Contextual are Contextualized Word Representations? Self-similarity, anisotropy, and the less-than-5% result.
• Howard and Ruder (2018). Universal Language Model Fine-tuning for Text Classification. ULMFiT and the fine-tuning recipe.
• Peters, Ruder, and Smith (2019). To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. Feature extraction versus fine-tuning, compared carefully.
• McCann et al. (2017). Learned in Translation, Contextualized Word Vectors. CoVe, contextual vectors from a translation encoder.
• Jurafsky and Martin. Speech and Language Processing, Chapter 14, RNNs and LSTMs. Bidirectional RNNs and stacked RNNs in textbook form.

Watch next

▶

▶