Static word embeddings changed NLP. For years, initializing a model with word2vec or GloVe vectors was the single easiest way to improve almost any NLP system. But they have structural flaws that no amount of data can fix. Understanding those flaws precisely is the best way to understand why the field moved to contextual models, attention, and the Transformer. Every crack in this section is a door to a later chapter.

Crack 1. One word, one vector, many meanings

"bank" (finance) and "bank" (river) are different words that happen to be spelled the same. A static table has one row for the string bank, so both senses must share a single vector $\mathbf{v}_{\text{bank}} \in \mathbb{R}^d$. What does training do with that conflict?

Arora et al. (2018) showed, theoretically and empirically, that the vector of a polysemous word is approximately a weighted average of its sense vectors, weighted by how often each sense occurs,

where $f$ counts occurrences. In plain words, the shared vector sits on the line between the two sense vectors, closer to whichever sense is more common. So the dominant sense wins, and the rare sense is nearly invisible. Let's check this with an experiment where we control the sense frequency. We generate a corpus in which "bank" appears with finance words a fraction p of the time and with river words otherwise, and train PPMI and SVD vectors (§1.4) for each p.

The single "bank" vector slides between the two senses in proportion to their frequency. At 90% finance usage its cosine with the finance words is 0.47 against 0.12 for the river words, at 50% the two are about equal, and at 10% the numbers swap to 0.12 and 0.45. In real English, the river sense is rare, so a static "bank" vector is essentially a finance word, and every downstream model that uses it is confused whenever someone goes fishing.

import numpy as np

def sense_weights(v, S):
    """v: (d,), S: (m, d) -> alpha: (m,) summing to 1"""
    raise NotImplementedError

Crack 2. No context, no composition

Even for words with one meaning, a static vector can't adapt to its sentence.

• "The movie was not good" and "The movie was good" contain nearly the same words. Averaging their word vectors gives nearly the same sentence vector.
• "The dog bit the man" and "The man bit the dog" have identical bags of words, so any order-free pooling gives them exactly the same vector.
• "bright" means different things in "bright student" and "bright light".

Meaning is compositional and order-dependent, and a lookup table is neither. You can stack a recurrent network on top of static vectors (that was the state of the art around 2016), but it squeezes the whole sentence through one sequential bottleneck. Chapter 3 shows why that struggles, and Chapter 4 shows how attention solves it.

Crack 3. Unseen and rare words

A word-level table has no row for a word it never saw, such as a new name, a typo ("embeddigns"), or a rare inflection ("unembeddable"). The standard hack, the <unk> token of §1.1, maps them all to a single vector, which throws away everything.

Rare words that were seen get noisy vectors, and you can put a number on the noise. A co-occurrence count $n$ behaves roughly like a Poisson variable, so its log (the quantity PMI and GloVe work with) has standard deviation about $1/\sqrt n$. A pair seen 4 times has its log count known only to about $\pm0.5$, while a pair seen 400 times is known to about $\pm0.05$. A word with a handful of occurrences is built from a handful of such noisy numbers. fastText's character n-grams (§1.5) help by sharing statistics across words. The real fix is to change what the vocabulary is made of, using subword tokens that can spell any string. That's Chapter 2.

Crack 4. Frequency and anisotropy

Here is a subtler problem. If you take trained embeddings and compute the cosine between random pairs of words, you'd hope for something near 0, like the random vectors of §1.3, where the average is 0 and the spread is $1/\sqrt d$. Instead it's usually clearly positive. All the vectors lean toward a common direction, so they occupy a narrow cone instead of spreading over the sphere. This is called anisotropy.

Mu and Viswanath (2018) found that word embeddings have a large shared mean vector, plus a few dominant directions that mostly encode word frequency rather than meaning. Those directions inflate every cosine and distort neighbor rankings (frequent words look similar to each other for being frequent). Mimno and Thompson (2017) found a related oddity in SGNS, where word vectors and context vectors point into opposite cones. Mu and Viswanath's remedy, "all-but-the-top", is simple post-processing in two steps.

1. Subtract the mean vector, $\tilde{\mathbf v}_w = \mathbf v_w - \bar{\mathbf v}$, where $\bar{\mathbf v}$ is the average of all word vectors.
2. Remove the projections onto the top $D$ principal components $\mathbf{a}_1,\dots,\mathbf a_D$ (unit vectors) of the centered vectors, $\mathbf v'_w = \tilde{\mathbf v}_w - \sum_{i=1}^{D}(\mathbf a_i^\top\tilde{\mathbf v}_w)\,\mathbf a_i$.

In plain words, take away what every word has in common, then take away the few directions along which words differ mostly by how frequent they are. They used $D\approx d/100$, so 3 directions for 300-dimensional vectors, and it consistently improved word-similarity benchmarks. Ethayarajh (2019) later found that contextual embeddings from BERT and GPT-2 are even more anisotropic, and in GPT-2's last layers the representations of two random words have an average cosine close to 1. That is why raw cosine similarity between Transformer hidden states can be misleading, and why sentence-embedding models are trained specifically to fix it (§7.3).

In this simulation the meaningful parts of the vectors are pure noise, so their true average cosine is 0. The shared offset alone lifts it to 0.83. Centering brings it back to 0.01, and removing the top two directions also kills the frequency signal in the vector lengths (the correlation between norm and log frequency drops from 0.96 to about 0).

import numpy as np

def all_but_the_top(E, D):
    raise NotImplementedError

Whitening, the stronger cure

All-but-the-top removes a few directions. Whitening goes further and makes the covariance of the embeddings the identity, so no direction is privileged at all. §0.10 builds it from the eigendecomposition of the covariance (Exercise 0.10d implements both the PCA and the ZCA versions). In one line, rotate the centered vectors onto their principal axes and rescale every axis to unit variance, so the cone becomes a round ball and the average cosine between random words drops to about 0. Su et al. (2021) showed that whitening BERT sentence embeddings this way improves their similarity rankings considerably, with no training. Keeping only the top $k$ principal axes also reduces dimension, and it often helps, because whitening blows the low-variance directions, which are sometimes mostly noise, up to unit size.

Crack 5. Embeddings absorb the biases of their data

Embeddings learn the statistics of text, and text reflects the society that wrote it. Bolukbasi et al. (2016) found that in word2vec trained on Google News, the vector $\mathbf v_{\text{he}} - \mathbf v_{\text{she}}$ defines a gender direction, and occupation words line up along it, with homemaker, nurse, receptionist at one end and maestro, captain, architect at the other. These aren't bugs in the algorithm. They're faithful measurements of the training data, and they matter as soon as embeddings feed decisions such as search ranking, résumé screening, or content moderation.

Try it on a real modern model (pick the "gender" preset).

Measuring bias with WEAT

Caliskan, Bryson and Narayanan (2017) turned this into a statistical test modeled on the Implicit Association Test from psychology, the Word Embedding Association Test (WEAT). Take two sets of target words $X$ and $Y$ (say, flowers and insects) and two sets of attribute words $A$ and $B$ (pleasant and unpleasant words). For one word $w$, its association is

and the effect size compares the two target sets in units of the spread of associations,

In plain words, $s$ says how much closer a word sits to the $A$ words than to the $B$ words, and $d$ says how many standard deviations apart the two target groups are on that scale, like Cohen's $d$. Significance comes from a permutation test. Shuffle which targets are called $X$ and which $Y$ many times, and ask how often a shuffled split gives a test statistic $\sum_{x}s(x) - \sum_{y}s(y)$ at least as large as the real one. Caliskan et al. reproduced a long list of documented human associations this way, for example flowers versus insects with pleasant versus unpleasant words, with an effect size of 1.50 in GloVe.

import numpy as np

def weat_effect_size(X, Y, A, B):
    raise NotImplementedError

def weat_pvalue(X, Y, A, B, n_perm=2000, seed=0):
    raise NotImplementedError

Debiasing, and why it's harder than it looks

The hard-debiasing method of Bolukbasi et al. has three steps.

1. Identify a bias direction $\mathbf g$, for example the top principal component of differences like $\mathbf v_{\text{he}}-\mathbf v_{\text{she}}$, $\mathbf v_{\text{man}}-\mathbf v_{\text{woman}}$, and so on.
2. Neutralize. For words that should be gender-neutral (occupations), remove the component along $\mathbf g$, giving $\mathbf v' = \mathbf v - (\mathbf v^\top\hat{\mathbf g})\,\hat{\mathbf g}$, with $\hat{\mathbf g} = \mathbf g/\lVert\mathbf g\rVert$.
3. Equalize. Make definitional pairs (he and she) symmetric around the neutral words.

import numpy as np

def neutralize(E, g, rows):
    raise NotImplementedError

The way forward, embeddings that read the sentence

Every crack points the same way. A word's representation should depend on its context. That's the leap from static to contextual embeddings.

• ELMo (Peters et al., 2018) ran a bidirectional LSTM language model over the sentence and used its hidden states as context-dependent word vectors.
• BERT (Devlin et al., 2019) and GPT use Transformers. Each layer lets every token look at every other token through attention and update its vector. After a few layers, the vector for "bank" in "the river bank flooded" is genuinely different from the one in "the bank raised rates".

But notice what doesn't change. The very first layer of every Transformer is the lookup table from §1.2, H = E[X], a static embedding per token. Everything in this chapter (the geometry, the training dynamics, the scatter-add gradient, the anisotropy, the bias) still applies to it. The layers above take those static vectors as raw material and contextualize them.

So the road ahead runs through three chapters, and a fourth on the social side.

1. Chapter 2, Tokenization, asks what exactly goes into the table. Subword pieces fix the unseen-word problem.
2. Chapter 3, From static to contextual meaning, shows how language models produce contextual vectors, and why recurrent encoders hit a wall.
3. Chapter 4, Attention, builds the mechanism that lets each vector read its context, from the dot products you now know intimately.
4. Chapter 19, Alignment and safety, returns to bias and harm at the level of whole systems.

One question is left hanging, and it is the right one to carry into Chapter 3. "Read the sentence" is not a loss you can minimize. Word2vec learned static vectors from a training task, predicting neighbors. What training task would force a model to work out which bank a sentence is about, and to say so in its vectors? Before that, §1.10 steps back and shows where all of these vectors end up in modern AI.

Read more

• Arora et al. (2018). Linear Algebraic Structure of Word Senses, with Applications to Polysemy. Senses as a linear superposition.
• Mu & Viswanath (2018), All-but-the-Top, and Mimno & Thompson (2017), The strange geometry of skip-gram with negative sampling.
• Ethayarajh (2019). How Contextual are Contextualized Word Representations? Anisotropy in BERT and GPT-2.
• Su et al. (2021). Whitening Sentence Representations for Better Semantics and Faster Retrieval. Whitening as a training-free fix.
• Bolukbasi et al. (2016), Man is to Computer Programmer as Woman is to Homemaker?, and Caliskan et al. (2017), Semantics derived automatically from language corpora contain human-like biases. The gender direction and WEAT.
• Gonen & Goldberg (2019). Lipstick on a Pig. Why debiasing by projection hides bias without removing it.

Watch next

▶

▶