So far every embedding has lived in a table. Item number $i$ gets row $i$, and that's it. This works beautifully when the world comes in a fixed vocabulary of IDs (words, users, products, graph nodes). But most of what we want to understand has no fixed vocabulary. There are infinitely many possible photographs, sentences, recordings, and video clips. You can't give every photo on Earth its own row.

The solution is the most important generalization in this chapter. Instead of looking up a vector, we compute it with a neural network called an encoder,

where $x$ is a raw input, $\theta$ are the network's parameters, and $\mathbf{z}$ is the input's embedding. In plain words, an encoder is any learned function that turns pixels, characters, or a waveform into a vector. Everything you learned about geometry (§1.3), training signals (§1.4 and §1.5), evaluation, and visualization (§1.6) carries over unchanged. Only the source of the vector is new.

There's a beautiful link between the two. Every encoder starts with a lookup or a projection. A text encoder begins with a token table exactly like §1.2. An image encoder begins by projecting small pixel patches with a matrix. So the encoder doesn't replace the embedding table. It builds on top of it, turning simple input embeddings into rich, context-aware ones.

The layer before the answer is an embedding

Here's a fact that surprises many people. Take any network trained for a task, chop off its last layer, and the activations that remain form an embedding space where inputs the task considers similar are close together. This is why "use the penultimate layer as features" became standard practice in computer vision (Razavian et al., 2014). A network trained to classify images learns, along the way, a space where cats cluster with cats.

Why must this happen? The last layer of a classifier is linear, $\text{logits} = \mathbf{W}\mathbf{z} + \mathbf{b}$, followed by a softmax. To classify well, the penultimate vectors $\mathbf{z}$ of different classes must be linearly separable, and vectors of the same class must land on the same side of every decision boundary. The network therefore has every incentive to bend the input space until classes form separate, roughly convex regions.

Let's watch it happen. Below, a tiny two-layer network (an MLP like the one in §0.7) learns to classify points from three intertwined spirals. We give its last hidden layer only two units so we can plot it. Before training, the classes are hopelessly tangled. In the learned 2-D hidden space, they come apart so that straight lines can separate them.

This is the whole story of representation learning in one picture. The network learns a space where the task becomes easy. The task decides what "similar" means. A classifier's space groups by class, a contrastive model's space groups by pairing, and a language model's space groups by what predicts the next token. It also explains the limits of transfer. Features learned for one task only help on another to the extent that the two tasks care about the same distinctions.

Images

The convolutional route

For a decade, image embeddings meant convolutional neural networks (CNNs, covered in §0.8). A CNN slides small learned filters across the image. Early layers detect edges and color blobs, middle layers detect textures and parts, and late layers respond to whole objects (Zeiler & Fergus, 2014). The final feature map is averaged over all spatial positions (global average pooling) to give one vector per image. For the classic ResNet-50, a $224\times224$ image ends as a $7\times7\times2048$ feature map, and pooling gives a 2,048-dimensional embedding.

Two design ideas are baked into a CNN, and they're worth naming because they are exactly what Vision Transformers give up.

• Locality. Each filter only looks at a small neighborhood, so nearby pixels interact first.
• Translation equivariance. The same filter is applied everywhere, so a cat in the corner is detected like a cat in the center.

These inductive biases make CNNs data-efficient. They are also constraints.

The Transformer route treats patches as words

In 2020, Dosovitskiy et al. asked a provocative question. What if we treat an image like a sentence? Their Vision Transformer (ViT) cuts an $H\times W$ image with $C$ color channels into a grid of $P\times P$ patches, flattens each patch into a vector $\mathbf{x}_i \in \mathbb{R}^{P^2 C}$, and projects it with a learned matrix,

where $\mathbf{p}_i \in \mathbb{R}^d$ is a learned position embedding for patch slot $i$. In plain words, each patch becomes a token embedding, and the matrix $\mathbf{W}$ plays the role of the embedding table for continuous inputs. Multiplying by $\mathbf{W}$ is a "soft lookup", a weighted blend of the matrix's columns with the pixel values as weights, just as multiplying by a one-hot vector was a hard lookup in §1.2.

Worked numbers for ViT-Base. A $224\times224$ RGB image with $16\times16$ patches becomes $14 \times 14 = 196$ patches of $16\cdot16\cdot3 = 768$ raw values each, projected to width $d = 768$. The projection has $768 \times 768 + 768 \approx 590$ thousand parameters, tiny next to the 86 million of the whole model. ViT prepends one extra learned "class" token, so the Transformer reads 197 tokens. With $14\times14$ patches instead (ViT-L/14, used by many CLIP models), the same image becomes 256 patches, and at $336\times336$ it becomes 576. A Transformer then processes the sequence exactly as it would process words.

Why did this win, when CNNs have such helpful built-in biases? Because with enough data, fewer assumptions scale better. ViT trained only on ImageNet's 1.3 million images loses to good CNNs. Pretrained on hundreds of millions of images, it wins. The model learns the useful biases on its own and isn't limited by the ones we hard-coded. Today's strongest image embedders (CLIP, SigLIP, DINOv2) are mostly Transformers, though CNNs remain excellent when data or compute is limited. The full story, from patches to how ViTs are trained, belongs to the vision part of the book.

import numpy as np

def patchify(img, P):
    """img: (H, W, C) -> (N, P*P*C), patches in raster order"""
    raise NotImplementedError

def unpatchify(patches, H, W, C, P):
    """(N, P*P*C) -> (H, W, C)"""

Text, at three levels

When people say "text embedding", they can mean three quite different things. Keeping them apart avoids a lot of confusion.

1. Token embeddings. These are the rows of a model's input table, exactly §1.2, one static vector per token. Every text model starts with one.
2. Contextual token embeddings. Now pass the whole sequence of token vectors through an encoder in which each position can read the others and update its own vector. The vector for "bank" then depends on the sentence around it, which is precisely what a static table cannot do (§1.9 measures the damage). Chapter 3 builds the first such encoders from recurrent networks, and Chapter 4 and Chapter 5 build the Transformer, the encoder inside every modern language model.
3. Sentence or document embeddings. These are one vector for a whole passage, made by pooling the contextual token vectors. The simplest and most common choice is to average them (mean pooling). Some models instead train one reserved position to hold a summary, such as BERT's [CLS] token (§7.1). Models that read strictly left to right often take the vector at the last position, because it is the only one that has read the whole text (§7.3). This one-vector-per-passage kind is what semantic search uses.

Here's the catch. A model trained only to fill in missing words, like BERT, does not automatically give good sentence vectors when you average its token vectors. Reimers and Gurevych (2019) found such averages worse than averaging static GloVe vectors on semantic similarity benchmarks, partly because of the anisotropy you'll meet in §1.9. Good sentence embedders are fine-tuned for the job. Paraphrase pairs are pulled together, and the other sentences in the batch are pushed apart, the same "real pair against in-batch negatives" recipe as the two-tower models of §1.7. Sentence-BERT (Reimers & Gurevych, 2019) and SimCSE (Gao et al., 2021) made this standard, and modern embedding models are trained the same way at much larger scale (§7.3).

import numpy as np

def mean_pool(H, mask):
    raise NotImplementedError

Sound

Audio arrives as a waveform, a long list of air-pressure samples, typically 16,000 per second for speech. Two common front ends turn it into a sequence of frame embeddings.

• Spectrogram first. Slice the waveform into short overlapping windows, take the Fourier transform of each to get energy per frequency band, and squash the frequencies onto the perceptual mel scale. Whisper (Radford et al., 2022) uses 25 ms windows every 10 ms with 80 mel bands, so 30 seconds of audio becomes a $3000\times80$ log-mel spectrogram. Two convolution layers (the second with stride 2) turn that into 1,500 frame embeddings, one per 20 ms, which a Transformer then reads.
• Learned from the raw wave. wav2vec 2.0 (Baevski et al., 2020) lets a stack of 1-D convolutions learn its own front end directly from the samples. Its total stride is 320 samples, so it produces one vector every 20 ms (50 per second) with a receptive field of about 25 ms. It's trained self-supervised by predicting masked frames contrastively, which is the word2vec idea again with time slices in place of words.

Pool the frames and you get an utterance-level embedding. Speaker embeddings that identify who is talking are built this way. Quantize the frames into a discrete codebook and you get audio tokens that can be embedded with a table and modeled exactly like words. That's how modern text-to-speech and voice models work, as §1.10 sketches and the neural audio codecs of the audio part of the book build.

Video

A video is a stack of images over time, and the obvious approaches have obvious costs.

• Per-frame embeddings. Embed each frame with an image encoder and pool or sequence them. It's cheap and simple, but it ignores motion inside each step.
• Spacetime patches (tubelets). Extend ViT's patches through time. VideoMAE (Tong et al., 2022) uses blocks of 2 frames × 16 × 16 pixels as tokens. Now motion is inside the token, but the token count explodes. Sixteen frames at 224×224 make $8 \times 14 \times 14 = 1{,}568$ tokens, eight times an image's 196, and the Transformers that read such tokens do work that grows with the square of the count (§4.7), so about 64 times more.

Video is where the question "what should the embedding predict?" becomes acute. Predicting every future pixel is wasteful, because the exact pattern of leaves in the wind doesn't matter. V-JEPA (Bardes et al., 2024) instead predicts the embeddings of masked spacetime regions from the embeddings of visible ones, so it can learn what's predictable and ignore what isn't. That idea, prediction in embedding space, is the root of learned world models that imagine the consequences of actions.

Two encoders, one space

Now the most consequential idea of modern multimodal AI. CLIP (Radford et al., 2021) trains an image encoder $f$ and a text encoder $g$ together on 400 million (image, caption) pairs from the web, so that an image and its caption land at the same point in a shared space.

You already know the training recipe, because it is the in-batch softmax of the two-tower recommenders in §1.7, with images in place of users and captions in place of items. Take a batch of $N$ pairs. Embed them as $\mathbf{u}_i = f(\text{image}_i)$ and $\mathbf{v}_j = g(\text{caption}_j)$, normalize every embedding to unit length, and compute every image-to-caption similarity, $s_{ij} = \hat{\mathbf{u}}_i^\top\hat{\mathbf{v}}_j / \tau$, where $\tau$ is a learned temperature (CLIP starts it at 0.07). The correct pairs sit on the diagonal of this $N\times N$ matrix, and every other entry is a mismatched pair. The loss asks each image to pick its own caption out of the batch, and each caption to pick its own image,

In plain words, this is cross-entropy over the rows plus cross-entropy over the columns of the similarity matrix, with the diagonal as the right answer. Real pairs are pulled together and the other $N - 1$ items in the batch serve as negatives, which is negative sampling from §1.5 with a softmax in place of independent sigmoids. This family of losses is called InfoNCE (van den Oord et al., 2018). §7.2 derives it properly, with its gradient, what it estimates, and how the temperature and batch size shape it, and CLIP itself gets a whole stretch of the book later, from its training to the sigmoid losses that followed.

Two numbers make the design concrete. At initialization all similarities are about equal, so each row's softmax is nearly uniform and the loss is about $\log N$. CLIP trained with $N = 32{,}768$, so its loss started near $\log 32768 \approx 10.4$, with 32,767 negatives for every image. And the temperature sets how much a cosine gap matters. With $\tau = 0.07$, a caption whose cosine is 0.1 higher than a rival's gets a logit $0.1/0.07 \approx 1.43$ higher, so it is favored by a factor of $e^{1.43} \approx 4.2$.

The payoff is zero-shot classification. To classify an image into classes the model never trained on, embed sentences like "a photo of a dog" and "a photo of a cat", and pick the one closest to the image embedding. Wording matters. The CLIP paper reports that averaging the text embeddings of 80 prompt templates per class ("a photo of a big {label}.", "a drawing of a {label}.", and so on) improved ImageNet accuracy by 3.5 points over a single default prompt. Here's the same trick with a text model running in your browser.

import numpy as np

def zero_shot(img, txt):
    """img: (N, d), txt: (C, T, d) -> (pred (N,), protos (C, d) with unit rows)"""
    raise NotImplementedError

What "embedding" means in retrieval

For practitioners, "embeddings" most often means retrieval. You store a large collection of passages, products, images, or code, and you want the most relevant ones for a query. The standard pipeline, called a bi-encoder because queries and documents are embedded separately, has four steps.

1. Split documents into chunks, embed each chunk once, and normalize.
2. Store the vectors in an approximate nearest neighbor index. You built the simplest one, an LSH index, in §1.3, and §7.4 builds the graph and quantization indexes used in production.
3. At query time, embed the query with the same model, retrieve the top $k$ by cosine, and optionally rerank those few with a slower, more accurate model that reads the query and each document together (§7.5).
4. Hand the results to a user, or to a language model that writes an answer from them. That last step is retrieval-augmented generation (RAG).

The whole picture so far

In PyTorch

In PyTorch the patch embedding is a strided convolution, and masked mean pooling is two lines.

With a pretrained network, an image embedding is "drop the classification head". This needs downloaded weights, so it is shown rather than run.

Read more

• Razavian et al. (2014). CNN Features off-the-shelf. The penultimate layer as a general image embedding.
• Dosovitskiy et al. (2021). An Image is Worth 16x16 Words. The Vision Transformer.
• Reimers & Gurevych (2019), Sentence-BERT, and Gao, Yao & Chen (2021), SimCSE. Contrastive sentence embeddings.
• van den Oord, Li & Vinyals (2018), Contrastive Predictive Coding, and Radford et al. (2021), CLIP. InfoNCE and the shared image and text space.
• Baevski et al. (2020), wav2vec 2.0, and Radford et al. (2022), Whisper. Audio frames as embeddings.
• Tong et al. (2022). VideoMAE. Tubelet embeddings for video.
• Malkov & Yashunin (2016), HNSW, Douze et al. (2024), The Faiss library, and Kusupati et al. (2022), Matryoshka Representation Learning.
• Muennighoff et al. (2022). MTEB. The standard benchmark for text embedding models.

Watch next

▶

▶

▶

▶