How a computer can know that “cat” is closer to “kitten” than to “carburetor”. This is the idea under every modern AI model, built from first principles and taken to interview depth.

Here is a question that sounds too simple to be interesting. How do you hand the word "cat" to a computer?

A neural network is, in the end, a big pile of multiplications and additions. It eats numbers and produces numbers. But most of what we care about isn't numbers. It's words, users, products, songs, amino acids, chess moves, zip codes. Before any learning can happen, every one of these symbols has to become a list of numbers, a vector.

How we choose those vectors matters enormously. This chapter is about the best answer we have found, the embedding, and every model in the rest of this book, from GPT to CLIP to Whisper to V-JEPA, starts with one.

To appreciate the good answer, we first need to feel the pain of the obvious ones. We'll try three, and each will fail in an instructive way. (If the linear algebra or the gradients below feel rusty, Chapter 0 has everything this chapter uses.) But first we have to say what "the word cat" even is to a program.

First, text becomes a list of IDs

Text arrives as one long string of characters. Before we can attach a vector to "cat", we have to decide what the units are and give each unit a name the computer can index. Three terms do this work for the rest of the book.

• A token is one unit of text that the model reads. In this chapter a token is a word or a punctuation mark, found by splitting the text on spaces.
• The vocabulary $\mathcal{V}$ is the set of distinct tokens the system knows, and $V$ is its size.
• A token's ID is its position in the vocabulary, an integer from $0$ to $V-1$. It is a name, like a seat number, and it says nothing about the token itself.

Here is the whole pipeline on a corpus of two sentences.

Fourteen tokens, eight distinct ones, so $V = 8$, and "cat" is ID 1 only because it sorts second, right after the period. Everything downstream sees just the array of IDs. The string "cat" never enters the network. Its ID does, and the question of this chapter becomes which vector should we attach to each ID?

Splitting on spaces is a toy, and it breaks quickly. "cat" and "cats" get unrelated IDs, "Cat" with a capital is a third token, and a word that never appeared while we built the vocabulary has no ID at all. Systems that split by words reserve a special <unk> token ("unknown") for those. Real language models split text into subword tokens instead, pieces such as "embed" and "dings" chosen by an algorithm so that any string at all can be spelled, with vocabularies of tens of thousands to a few hundred thousand tokens. Chapter 2 builds those tokenizers from scratch. Nothing in this chapter depends on how the IDs were produced, so we'll use whole words, which are easier to read.

import numpy as np
from collections import Counter

def build_vocab(tokens, min_count=1):
    """-> list of strings: ["<pad>", "<unk>", most frequent token, ...]"""
    raise NotImplementedError

def encode(tokens, vocab):

Attempt 1, just number them

We now have an integer for every word. The simplest idea is to hand that integer to the model directly.

IDs are perfect for storing and indexing words, which is what they were made for. But they are a terrible way to feed words into a model, because the model treats the IDs as quantities. To a linear model, the input 2 is twice the input 1, and 1 sits exactly halfway between 0 and 2.

That last fact has a sharp consequence. Any linear function $f(\text{id}) = a\cdot\text{id} + b$ satisfies $f(1) = \tfrac12\big(f(0) + f(2)\big)$, whatever $a$ and $b$ are. So a linear model fed raw IDs is forced to predict that a dog's score on anything (cuteness, price, sentiment) is the average of a cat's and a carburetor's. No amount of training can change that, because it's baked into the input.

The numbers impose an order and a magnitude that don't exist. Whatever arbitrary order we picked (alphabetical, by frequency, at random) gets baked into the model as if it meant something. A deep network can learn to undo some of this with enough units and data, but it has to spend capacity carving the number line back into separate pieces, one per word.

Attempt 2, one-hot vectors

The standard fix in classical machine learning is the one-hot encoding. If the vocabulary $\mathcal{V}$ has $V$ words, word number $i$ becomes a vector of length $V$ that is all zeros except for a single 1 in position $i$,

With $V = 4$, the four words become

This fixes the fake ordering. No word is "bigger" than another, and no word sits between two others. Each one just gets its own axis. Let's write it in code. (Press ▶ Run or Shift+Enter. The first run downloads Python into your browser, which takes a few seconds.)

The last line prints the Gram matrix, the table of every dot product $\mathbf{e}_i^\top\mathbf{e}_j$. It is the identity matrix $\mathbf{I}$. That one fact contains both the strength and the fatal weakness of one-hot codes.

import numpy as np

def one_hot(ids, V):
    """ids: int array of any shape -> float32 array of shape ids.shape + (V,)"""
    ids = np.asarray(ids)
    # your code here
    raise NotImplementedError

That -1 test is not pedantry. Negative indices silently wrap around in NumPy and PyTorch, so a bug that produces -1 IDs (a common "unknown token" sentinel) will quietly map them to the last word in your vocabulary. Here is the PyTorch version.

The fatal flaw, everyone is equally far from everyone

Now look at the geometry. For any two different words $i \neq j$,

In plain words, every pair of different words has a dot product of zero, sits exactly $\sqrt{2}$ apart, and is at a right angle to every other. (The distance follows from $\lVert \mathbf{e}_i - \mathbf{e}_j \rVert^2 = 1 + 1 - 2\cdot 0 = 2$.)

So "cat" is exactly as similar to "kitten" as it is to "carburetor". The representation contains no notion of similarity at all. Check it yourself.

Why this kills generalization

This isn't just ugly. It actively prevents learning from transferring. Feed a one-hot input into a linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{h\times V}$. The product is

so the layer only ever reads column $i$. The backward pass only ever writes column $i$ too. If the loss has gradient $\mathbf{g} = \partial\mathcal{L}/\partial(\mathbf{W}\mathbf{e}_i) \in \mathbb{R}^h$ at the layer's output, the gradient of a linear layer (derived in §0.2) is the outer product

which is zero everywhere except column $i$. In plain words, a training example about word $i$ can only change word $i$'s private column. Everything the model learns about "cat" is stored in the cat column, and the kitten column receives exactly zero gradient from cat examples.

Here is a two-row example you can check by hand. With

which is just the third column (the code below counts from 0, so carburetor is index 2 there). We can watch the consequence. Below, a tiny linear model learns a "cuteness score" from examples that never mention kitten. With one-hot inputs, its prediction for kitten stays at exactly the initial value.

What a linear model on one-hot features actually learns

We can say exactly what a linear model on one-hot features converges to, and the answer is revealing. Stack $N$ training examples into a matrix $\mathbf{X} \in \{0,1\}^{N\times V}$ whose row $n$ is $\mathbf{e}_{x_n}^\top$, with targets $\mathbf{y} \in \mathbb{R}^N$, and fit weights $\mathbf{w} \in \mathbb{R}^V$ by ridge regression,

Setting the gradient to zero gives the normal equations $(\mathbf{X}^\top\mathbf{X} + \lambda\mathbf{I})\,\mathbf{w} = \mathbf{X}^\top\mathbf{y}$. Now look at the two pieces. Entry $(i, j)$ of $\mathbf{X}^\top\mathbf{X}$ counts the examples that are both word $i$ and word $j$, which is impossible unless $i = j$. So $\mathbf{X}^\top\mathbf{X} = \operatorname{diag}(n_1, \dots, n_V)$, where $n_i$ is how many examples are word $i$. And entry $i$ of $\mathbf{X}^\top\mathbf{y}$ is $S_i$, the sum of the targets of word $i$'s examples. A diagonal system solves one coordinate at a time,

In plain words, each word's weight is its own average target, shrunk toward zero more strongly when the word is rare. The equations for different words never touch each other. A word never seen in training ($n_i = 0$, so $S_i = 0$) gets exactly $w_i = 0$, whatever we know about its neighbors.

A worked example. Suppose cat appears 3 times with targets 1.0, 0.8, 0.9, kitten once with 1.0, and $\lambda = 1$. Then $w_{\text{cat}} = 2.7/4 = 0.675$ and $w_{\text{kitten}} = 1.0/2 = 0.5$. Kitten's single glowing example is shrunk hard, and nothing about the three cat examples helps it. A fifth word with no examples gets 0. That is the "no transfer" problem written as a formula.

import numpy as np

def fit_onehot_ridge(ids, y, V, lam):
    """ids: (N,) ints in [0, V); y: (N,) floats -> w: (V,)"""
    raise NotImplementedError

The second flaw, size

Real vocabularies are large. Modern language-model tokenizers use $V \approx 32{,}000$ to $256{,}000$ tokens. Recommender systems embed hundreds of millions of users and items. A one-hot vector for a single token in Llama 3's vocabulary ($V = 128{,}256$) holds 128,256 numbers, of which 128,255 are zero.

Put numbers on a single training batch. Take $B = 8$ sequences of $T = 4{,}096$ tokens, so $N = 32{,}768$ token positions. Stored as one-hot float32 rows, the batch takes $32{,}768 \times 128{,}256 \times 4$ bytes $\approx 16.8$ GB, before the model has done anything. Stored as int64 IDs, it takes $32{,}768 \times 8$ bytes $\approx 262$ KB, which is 64,000 times less. Multiplying the one-hot batch by a $V\times d$ matrix with $d = 4{,}096$ costs $2NVd \approx 3.4 \times 10^{13}$ floating-point operations, and all but a vanishing fraction of them multiply by zero. Copying the right rows instead costs $Nd \approx 1.3\times 10^{8}$ number copies.

Play with the sizes below, and flip the left panel between one-hot and a small dense embedding.

Attempt 3, short codes that nobody learned

If one-hot vectors are too long, why not compress them? Write each ID in binary. With $V$ words you need only $\lceil \log_2 V \rceil$ bits, so Llama 3's 128,256 tokens fit in 17 bits instead of 128,256 slots. Hashing tricks do something similar, mapping each item to a short pseudo-random code.

These codes are short and dense, but now similarity comes back wrong. With 2 bits, cat = 00, dog = 01, carburetor = 10, kitten = 11. The Hamming distance (number of differing bits) says cat and dog are neighbors (distance 1), while cat and kitten are as far apart as possible (distance 2). Worse, a linear model on these bits shares weights between items that happen to share a bit. Every update for dog (01) also moves kitten (11), because they share the last bit. That is transfer, but transfer between arbitrary strangers.

import numpy as np

def binary_code(ids, V):
    """ids: int array of any shape -> float32 array of shape ids.shape + (n_bits,), MSB first"""
    raise NotImplementedError

The idea, short and dense learned vectors

Here is the leap. Keep the vectors short and dense, as in Attempt 3, but let the model learn the numbers instead of fixing them in advance. Give each word a short list of real numbers (say $d = 300$),

If training goes well, words used in similar ways end up with similar vectors. Then two things happen that none of our attempts could deliver.

1. Similarity becomes geometry. "cat" and "kitten" are close together, and "carburetor" is far away. We can measure meaning with a ruler (§1.3).
2. Learning transfers, between the right items. Every gradient update for "cat" nudges a vector that sits near "kitten", and any downstream weight that reads that region of space now treats kitten similarly. Rare words borrow strength from common neighbors.

This mapping from discrete things to dense vectors is called an embedding, and the vector $\mathbf{v}_{\text{cat}}$ is the embedding of "cat". Formally, we write

In plain words, an embedding is a function from the vocabulary to $d$-dimensional space, stored as a matrix $\mathbf{E}$ with one row per item. Item $i$'s vector is row $i$.

Two remarks keep this honest. First, $\mathbf{E}$ has exactly as many numbers as the weight matrix of a linear layer on one-hot inputs with $d$ outputs. The embedding is not a new kind of parameter. It is that same layer, stored row-wise and computed by indexing, as §1.2 proves. What changes the outcome is what sits above it. The layers that read the vectors are shared by every word, so once cat and kitten get nearby vectors, everything those shared layers learn about one applies to the other.

Second, dense codes are combinatorially richer than one-hot ones. A one-hot code with $d$ units can name only $d$ items. A code where each of $d$ units can be on or off can name $2^d$ patterns, which is about a million for $d = 20$, and real-valued units can do far more. Goodfellow, Bengio and Courville (2016, §15.4) make this argument for distributed representations in general.

A tool for the rest of the chapter

Before we leave one-hot land, one coding problem that we'll reuse constantly. Every similarity search, nearest-neighbor check, and clustering step in this chapter needs all pairwise distances between a set of vectors, fast.

import numpy as np

def pairwise_distances(X):
    """X: (n, d) -> (n, n) Euclidean distance matrix"""
    X = np.asarray(X, dtype=np.float64)
    # your code here
    raise NotImplementedError

The expansion trades a little accuracy for a lot of speed. When two points are very close compared with their distance from the origin, $\lVert\mathbf a\rVert^2 + \lVert\mathbf b\rVert^2$ and $2\mathbf a^\top\mathbf b$ are two huge, nearly equal numbers, and subtracting them loses most of the significant digits (catastrophic cancellation). Centering the data first (subtracting the mean vector, which leaves all distances unchanged) shrinks the norms and recovers most of the precision. torch.cdist (for larger inputs) and scikit-learn's euclidean_distances use the same expansion with a similar clamp.

Read more

• Hinton, McClelland & Rumelhart (1986). Distributed Representations. The original argument for patterns over one-unit-per-concept codes.
• Bengio et al. (2003). A Neural Probabilistic Language Model. The learned lookup table as the cure for one-hot language models.
• Goodfellow, Bengio & Courville (2016). Deep Learning, §15.4 Distributed Representation. Why distributed codes are exponentially more expressive than local ones.
• Jurafsky & Martin. Speech and Language Processing, Chapter 5 (Embeddings). The best textbook companion for this whole chapter.
• Guo & Berkhahn (2016). Entity Embeddings of Categorical Variables. Embeddings as the replacement for one-hot features in tabular models.

Watch next

▶