You've trained embeddings. Are they any good? What's actually in them? These questions are harder than they look, because embedding spaces are high-dimensional, and every way of summarizing them (a benchmark number, an analogy, a 2-D picture) throws information away. This section teaches the standard tools and, just as important, how each one can fool you.

Nearest neighbors, the first sanity check

The simplest probe is the one you built in §1.3. For a query word, list the words with the highest cosine similarity. If frog's neighbors are toad, frogs, lizard, salamander, training probably worked. If they're the, of, said, something is broken. Typical causes are missing subsampling, too few epochs, a forgotten scatter-add, or comparing raw dot products instead of cosines.

Neighbors are qualitative, but they catch most bugs, and they reveal what kind of similarity your setup learned, substitutable words or merely associated ones (§1.4).

Intrinsic evaluation asks whether cosines agree with people

Similarity benchmarks give word pairs rated by humans, such as (tiger, cat) = 7.35 or (king, cabbage) = 0.23 on a 0 to 10 scale. We compute the cosine for each pair and measure how well the two rankings agree, using Spearman's rank correlation $\rho$,

where $\operatorname{corr}$ is the ordinary Pearson correlation. In plain words, replace every score by its position in sorted order, then ask how linearly related the two lists of positions are. Ranks, not raw values, because human scales and cosine scales aren't comparable. Only the order matters, so any monotone transformation of the cosines leaves $\rho$ unchanged.

When there are no ties, $\rho$ has a closed form in terms of the rank differences $d_i$ over $n$ pairs, $\rho = 1 - \frac{6\sum_i d_i^2}{n(n^2-1)}$. A worked example with four pairs. Humans give $(9.1, 7.4, 3.0, 0.2)$ and the model gives cosines $(0.62, 0.70, 0.31, 0.05)$. The human ranks (lowest is 1) are $(4, 3, 2, 1)$, the model ranks are $(3, 4, 2, 1)$, the differences are $(1, -1, 0, 0)$, and $\rho = 1 - \frac{6\cdot2}{4\cdot15} = 0.8$. The model swapped the top two pairs and got everything else right.

Common benchmarks are WordSim-353 (Finkelstein et al., 2002), MEN (Bruni et al., 2014), and SimLex-999 (Hill et al., 2015).

Extrinsic evaluation plugs the embeddings into a real task (sentiment classification, named-entity recognition, retrieval) and measures task performance. That's what ultimately matters, and intrinsic scores don't always predict it. Chiu, Korhonen and Pyysalo (2016) found that word-similarity scores correlated poorly, and sometimes negatively, with performance on downstream tagging tasks.

import numpy as np

def spearman(x, y):
    raise NotImplementedError

Analogies, the famous trick and its fine print

The result that made word2vec famous is

The analogy a is to b as c is to ? ("man is to king as woman is to ?") is answered by the word whose vector is most similar to $\mathbf{v}_b - \mathbf{v}_a + \mathbf{v}_c$. With unit vectors, this 3CosAdd rule is

(The second form holds because, for unit vectors, the cosine with the target is the dot product with it divided by the target's length, which is the same for every candidate $x$.) In plain words, find the word that is similar to king and woman, and dissimilar to man.

A worked example with five hand-made 3-D vectors, the same ones the exercise uses. They are man $=(1, 0, 0.1)$, woman $=(0, 1, 0.1)$, king $=(1, 0.1, 1)$, queen $=(0.1, 1, 1)$, apple $=(0.3, 0.3, -1)$. After normalizing, the target $\hat{\mathbf v}_{\text{king}} - \hat{\mathbf v}_{\text{man}} + \hat{\mathbf v}_{\text{woman}}$ is about $(-0.29, 1.07, 0.71)$. Its cosines with the five words are man $-0.17$, woman $0.86$, king $0.28$, queen $0.94$, apple $-0.33$. Queen wins here even without excluding the inputs, but notice how close woman comes. In real embedding spaces the input word usually wins.

Geometrically, this works when a relation (male to female, country to capital, verb to past tense) corresponds to a roughly constant offset vector. Why would training produce such offsets? Here is a heuristic explanation via PMI (§1.5). If king and queen differ in their co-occurrence statistics in the same way that man and woman do (more he and his versus more she and her), then because $\mathbf{v}^\top\mathbf{u}\approx\operatorname{PMI}$, the vector differences align. Allen and Hospedales (2019) make this argument precise under explicit assumptions.

import numpy as np

def analogy(E, vocab, a, b, c, method="add", exclude=True):
    raise NotImplementedError

import numpy as np

def analogy_accuracy(E, quads, method="add"):
    """-> (fraction of questions answered correctly, predictions of shape (Q,))"""
    raise NotImplementedError

Looking at the space by projecting to 2-D

We can't see 300 dimensions, so we project to 2 (or 3). The methods differ in what they choose to keep, and that determines what they distort.

PCA, the best flat shadow

Principal component analysis finds the orthogonal directions of maximum variance and projects onto the top two. With centered data $\mathbf{X}$ (rows are words, the mean row subtracted) and SVD $\mathbf{X} = \mathbf{U}_{\text{svd}}\boldsymbol\Sigma\mathbf{V}_{\text{svd}}^\top$, the 2-D coordinates are $\mathbf{X}(\mathbf{V}_{\text{svd}})_{:,1:2} = (\mathbf{U}_{\text{svd}})_{:,1:2}\boldsymbol\Sigma_{1:2}$. That's the same Eckart–Young optimality as §1.4, the best rank-2 approximation in squared error. The fraction of variance kept is

where $\sigma_i$ are the singular values of the centered data. In plain words, each squared singular value is the variance along one principal direction (times the number of rows), so this ratio is the share of the total spread that the picture shows.

PCA is linear and faithful in one specific sense. Distances in the picture are never larger than the real ones, because an orthogonal projection can only shrink vectors. Large-scale structure (which clusters are far apart) is preserved as well as two dimensions allow. Its weakness is that if the top two directions only explain, say, 15% of the variance, most of the structure is invisible, and distinct groups can land on top of each other. Centering matters too. Without it, the first "component" mostly points at the mean vector, which for anisotropic embeddings (§1.9) is a large shared offset that says nothing about differences between words.

import numpy as np

def pca(X, k):
    """-> (Z (n, k), components (k, d), explained variance fractions (k,))"""
    raise NotImplementedError

t-SNE keeps the neighbors and sacrifices everything else

t-SNE (van der Maaten & Hinton, 2008) doesn't try to preserve distances at all. It tries to preserve who is near whom, in three steps.

1. In the original space, turn distances into neighbor probabilities, $p_{j\mid i} \propto \exp(-\beta_i\lVert\mathbf{x}_i-\mathbf{x}_j\rVert^2)$, normalized over $j \neq i$. The precision $\beta_i$ (written $1/2\sigma_i^2$ in the paper) is tuned per point so the distribution has a fixed perplexity $2^{H(P_i)}$, where $H(P_i) = -\sum_j p_{j\mid i}\log_2 p_{j\mid i}$ is the entropy in bits. Perplexity is roughly "the effective number of neighbors" (§0.5 explains why). Then symmetrize, $p_{ij} = (p_{j\mid i}+p_{i\mid j})/2n$, where $n$ is the number of points.
2. In 2-D, use a heavy-tailed Student-t kernel, $q_{ij} \propto (1+\lVert\mathbf{y}_i-\mathbf{y}_j\rVert^2)^{-1}$, normalized over all pairs.
3. Move the 2-D points $\mathbf{y}_i$ to minimize $\operatorname{KL}(P\,\Vert\,Q) = \sum_{ij}p_{ij}\log\frac{p_{ij}}{q_{ij}}$, by gradient descent with gradient

In plain words, pairs that are neighbors in the original space but too far apart in the picture ($p_{ij} > q_{ij}$) attract, pairs that are too close repel, and the force is damped for points that are already far apart.

The heavy tail solves the crowding problem. There isn't enough room in 2-D to place all of a point's moderately distant neighbors at faithful distances, so the t-kernel lets non-neighbors be pushed far away. That's what makes t-SNE clusters look so crisp. It also means sizes and gaps in a t-SNE plot are not to be trusted. Because each $\beta_i$ adapts to local density, a tight cluster and a diffuse one come out about the same size.

import numpy as np

def tsne_affinities(X, perplexity, n_steps=64):
    """-> (P_cond (n, n) with rows summing to 1, P (n, n) symmetric joint summing to 1)"""
    raise NotImplementedError

Wattenberg, Viégas and Johnson (2016) collected the lessons of reading t-SNE plots. Run the next cell to see three of them on synthetic data where we know the truth.

Here is what to notice.

• Cluster sizes mean nothing. Blue is 10 times tighter than orange in reality, but t-SNE draws them about the same size, because it adapts its precision to local density.
• Distances between clusters may mean nothing. The gap from blue to green (about 134 in the original space) is 10 times the gap from blue to orange (about 13), but t-SNE doesn't show that reliably.
• Hyperparameters change the story. Perplexity changes the picture, sometimes completely.

UMAP

UMAP (McInnes, Healy & Melville, 2018) builds a weighted $k$-nearest-neighbor graph, treats it as a fuzzy topological structure, and optimizes a 2-D layout whose graph matches it, using a cross-entropy loss and negative sampling (the same trick as §1.5). It's much faster than exact t-SNE, scales to millions of points, and is often said to preserve more global structure. But Kobak and Linderman (2021) showed that much of that difference comes from initialization. Both methods preserve global structure better when initialized with PCA. The same warnings apply. n_neighbors and min_dist change the picture, and sizes and gaps are unreliable.

import numpy as np

def knn_preservation(X, Y, k):
    raise NotImplementedError

In PyTorch and the ecosystem

PCA is a few lines of linear algebra, so it runs in PyTorch directly, on the GPU if you like.

For t-SNE and UMAP, use the standard libraries rather than writing your own optimizer.

Read more

• Hill, Reichart & Korhonen (2015). SimLex-999. Similarity versus relatedness.
• Chiu, Korhonen & Pyysalo (2016). Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance. Why benchmark scores are not the whole story.
• Levy & Goldberg (2014). Linguistic Regularities in Sparse and Explicit Word Representations. 3CosMul.
• Linzen (2016), Issues in evaluating semantic spaces using word analogies, and Nissim et al. (2020), Fair Is Better than Sensational. The fine print on analogies.
• van der Maaten & Hinton (2008). Visualizing Data using t-SNE. The original paper, including the crowding problem.
• Wattenberg, Viégas & Johnson (2016). How to Use t-SNE Effectively. Interactive lessons on reading t-SNE.
• Kobak & Linderman (2021). Initialization is critical for preserving global data structure in both t-SNE and UMAP.
• Coenen & Pearce. Understanding UMAP. A visual explainer from Google PAIR.

Watch next

▶

▶

▶

▶