Sentence-BERT was trained on about a million sentence pairs, with a batch of 16. Open any embedding leaderboard today and the top models were trained on hundreds of millions of pairs with batches of tens of thousands, follow a written instruction about the task, come in sizes from 20 million to many billions of parameters, and let you cut their vectors to a quarter of their length or store them in one bit per dimension. None of this changes the core idea of §7.2. It's the same InfoNCE loss. What changed is the data, the stages, the scale, and a set of engineering ideas that make embeddings cheaper to store and search.

This section walks through how a modern embedding model is built, then the three tricks you'll use in practice to make embeddings smaller or sharper (Matryoshka truncation, quantization, and late interaction), and finally how to read the benchmark everyone cites, MTEB.

The modern recipe has stages

The open models you'll meet most often, E5 (Wang et al., 2022), GTE (Li et al., 2023), and BGE (Xiao et al., 2023), differ in details but share a shape. Here it is, with the numbers each paper reports.

The weak pairs come from structure the web already has, a title and its body, a question and its accepted answer (Stack Exchange, Reddit), a paper's title and abstract, anchor text and the page it links to, a function and its docstring. They are noisy, so filtering matters. E5's consistency filter trains a model on the noisy pairs, then keeps a pair only if the passage ranks in the model's top 2 against a pool of a million random passages, which removed about 80 percent of the data.

Can stage 1 work with no pairs at all? Contriever (Izacard et al., 2022) showed that it can, which is worth knowing because it isolates what contrastive pretraining itself contributes. A positive pair is two random spans cropped independently from the same document, so they share a topic but rarely share exact sentences. Negatives come from a MoCo-style queue of 131,072 earlier passages embedded by a slowly updated copy of the encoder (§7.2), with $\tau = 0.05$. With no labels of any kind, it beat BM25 on 11 of 15 BEIR datasets in recall@100, though not in nDCG@10, where precise ranking at the top needs supervised data. Fine-tuned on MS MARCO afterward, it became a strong supervised retriever, and it is the unsupervised encoder that HyDE builds on in §7.7.

Why two contrastive stages instead of one? The weak data is huge and diverse but noisy, which teaches broad coverage. The labeled data is clean but narrow, which teaches precise relevance. Large batches matter in stage 1, where negatives are random and need to be numerous (E5's ablation from §7.2 showed 1K versus 32K batches is worth almost 6 points of nDCG). In stage 2, hard negatives already supply a strong gradient, so a large batch is unnecessary (GTE notes this explicitly, and its stage-1 batch-size study found gains saturating around ten thousand).

GTE adds one twist to the loss worth knowing, which it calls an improved contrastive loss. For the $i$-th query and document pair $(q_i, d_i)$ in the batch, with $s$ the cosine similarity and $\tau$ the temperature, the numerator is still $e^{s(q_i, d_i)/\tau}$, but the denominator sums four kinds of terms, query $i$ against every document $d_j$ (the usual InfoNCE row), query $i$ against every other query $q_j$, every query $q_j$ against document $i$ (the reverse direction, as in CLIP's column loss), and document $i$ against every other document $d_j$. All of these embeddings are already computed for the batch, so each pair sees roughly four times as many negatives at no extra encoding cost.

Prefixes and instructions

A query and the passage that answers it are different kinds of text. "symptoms of vitamin d deficiency" is a short question, and the answer is a paragraph of medical prose. A model that embeds both identically has to put them near each other anyway. Giving the encoder a hint about which is which makes the job easier, and models are trained with these hints, so you must use them at inference too.

• E5 prepends query: and passage: . Its authors recommend query: on both sides for symmetric tasks like paraphrase detection.
• BGE adds an instruction to the query side only, for retrieval, Represent this sentence for searching relevant passages: .
• LLM-based embedders like E5-Mistral (Wang et al., 2024) put a full natural-language task description on the query, as in Instruct: Given a web search query, retrieve relevant passages that answer the query followed by Query: ..., and leave documents unprefixed. That's a deliberate design. Documents are embedded once into the index, and the task is chosen at query time.

How much do instructions matter? In E5-Mistral's ablation, removing them dropped the average MTEB score from 64.5 to 60.3. Forgetting the prefix that a model was trained with is one of the most common silent bugs in retrieval systems. Nothing crashes, and quality quietly falls.

Embedders built from LLMs

Since 2024 many of the strongest embedding models are decoder LLMs fine-tuned contrastively. E5-Mistral fine-tuned Mistral-7B for fewer than a thousand steps with LoRA, which freezes the pretrained weights and trains only a small low-rank correction to each weight matrix (§14.5), on synthetic data plus 13 public datasets. Instruction-tuned chat models (GPT-4 and GPT-3.5-Turbo) wrote the synthetic examples, query and passage pairs with hard negatives for hundreds of thousands of embedding tasks in 93 languages. Other well-known designs are LLM2Vec (BehnamGhader et al., 2024) and NV-Embed (Lee et al., 2024).

Decoders bring one new problem, pooling. In a causal model, the hidden state at position $t$ has seen only tokens $1, \dots, t$. Mean pooling averages states that each saw a different prefix, and only the final token has seen the whole text. So the common choices are the following.

• Last-token pooling. Append an end-of-sequence token and take its final hidden state, as E5-Mistral does. In that paper's ablation, mean pooling was only slightly worse (64.1 against 64.5), so the choice matters less than you'd think, but last-token is the norm.
• Make attention bidirectional. LLM2Vec and NV-Embed remove the causal mask during contrastive training so every token sees every other, then mean-pool. NV-Embed also adds a small learned attention-pooling layer.

Last-token pooling has a classic implementation bug. With right padding, the last real token of each sequence is at a different position, and taking position $-1$ grabs a padding token. The fix is to index each row by its own length, or to pad on the left so that every sequence ends at the same position.

import numpy as np

def last_token_pool(H, mask):
    return H[:, -1]

Big embedders cost real money. A 7B-parameter model embeds a passage with roughly 60 times the FLOPs of a 110M-parameter encoder (the forward pass costs about $2N$ FLOPs per token for $N$ parameters, §5.6), and it emits 4,096-dimensional vectors, more than five times the storage of a 768-dimensional one. For a corpus of a billion chunks, that's the difference between a modest and a very large index. That's where the next three ideas come in.

Matryoshka embeddings, many sizes in one vector

Suppose you need 768 dimensions for the best quality, but most of your traffic would be fine with 64, and your index would be twelve times smaller. Can one model give both? Matryoshka Representation Learning (MRL, Kusupati et al., 2022) trains the embedding so that its prefixes are good embeddings too. With a set of nested sizes $\mathcal{G} = \{8, 16, 32, \dots, d\}$, the loss sums the usual loss over every prefix,

where $\mathbf{z}_{1:m}$ is the first $m$ coordinates of the embedding, $\mathcal{L}$ is the base loss (cross-entropy for classification, InfoNCE for retrieval, applied after renormalizing the prefix), and $c_m$ are weights, all 1 in the paper. The paper used $\mathcal{G} = \{8, 16, \dots, 2048\}$ for 2,048-dimensional ResNet features.

In plain words, the model is graded at every size at once, so it learns to put the most important information first, like nested dolls.

The paper reports up to 14 times smaller embeddings at the same ImageNet accuracy, and up to 14 times faster large-scale retrieval with adaptive retrieval, where you shortlist candidates with a short prefix and rerank them with the full vector. Many current text embedders are trained this way, and some APIs expose the prefix length as a parameter.

Why doesn't an ordinary model have this property? Because nothing in its loss rewards putting information in the first coordinates. Any rotation of an embedding space gives exactly the same cosines, so a model is free to spread information evenly across all dimensions, and generally does. Here is the effect in a setting where we control everything. We make data whose information lives mostly in a few directions, then store it two ways, once with coordinates sorted by importance (what MRL, or PCA, achieves) and once randomly rotated (what an ordinary model gives you). Both have identical cosines at full length.

At full length the two are identical, as they must be. Truncated, the sorted version barely notices, while the rotated one collapses. For linear features, sorting directions by variance (PCA) gives the best nested subspaces in the least-squares sense, so you can get Matryoshka-like behavior after the fact. For a deep model, MRL's loss is what makes the network produce sorted features in the first place, and the paper found that post-hoc compression like SVD lost much more accuracy at small sizes.

import numpy as np

def truncate(E, m):
    return E[:, :m]

def mrl_infonce(Q, P, dims, tau):
    raise NotImplementedError

Quantized embeddings, int8 and one bit

The other way to shrink an index is to store fewer bits per dimension. A float32 vector of 1,024 dimensions is 4 KB. A hundred million of them are 410 GB, more RAM than most single servers have, and memory is the expensive part of a vector index.

Scalar (int8) quantization. Map each dimension's range to 256 levels. With a minimum $\ell_j$ and maximum $u_j$ for dimension $j$, measured on a calibration sample of the corpus,

where $c_j \in \{0, \dots, 255\}$ is the stored byte and $\tilde z_j$ the reconstruction. In plain words, slice each dimension's range into 255 equal steps and store the step number. For values inside the calibration range the error per coordinate is at most $s_j/2$. A value outside it (a document more extreme than anything in the calibration sample) is clipped to 0 or 255 first, and its error can be larger, which is why the calibration sample should be large and representative. That's 4 times smaller than float32, and int8 dot products are fast on modern CPUs and GPUs. The general theory of quantization (per-channel scales, outliers, error analysis) is a story told for model weights in the basics of quantizing LLMs, and it applies here unchanged.

Binary quantization. Keep only the sign of each coordinate, one bit per dimension, 32 times smaller than float32. Similarity becomes the Hamming distance, the number of differing bits, which is one XOR and one population count per 64 dimensions, about the cheapest comparison a CPU can do. Binary codes lose a lot of precision, so the standard trick is rescoring. Search the binary index for, say, 4 to 10 times more candidates than you need, then rescore those few with a more precise representation, either the float query against the stored binary or int8 documents, or the original float vectors kept on disk.

Let's measure all three on synthetic clustered embeddings, using recall of the exact top 10, which is a strict test.

Three lessons. Int8 is nearly free. Binary alone is rough, getting fewer than half of the exact top 10 right. Rescoring a binary shortlist recovers most of the loss, and the larger the shortlist, the better, at a small extra cost. Real embedding models usually do better with binary codes than this synthetic data suggests, especially at high dimensions, and downstream metrics like nDCG@10 are more forgiving than recall of the exact neighbors. Hugging Face's study of binary and scalar embedding quantization (Shakir, Aarsen & Lee, 2024) found that binary search with rescoring kept most of the retrieval quality of float32 for large models. Measure it on your own data.

import numpy as np

def binary_search(Q, D, k, rescore):
    raise NotImplementedError

The two ideas compose. A 1,024-dimensional Matryoshka model truncated to 256 dimensions and binarized takes 32 bytes per vector, 128 times less than the float32 original. A hundred million vectors then take 3.2 GB.

ColBERT and late interaction

Every method so far squeezes a whole passage into one vector, and a single vector has to compromise. A 300-word passage covers several facts, and a query usually asks about one. ColBERT (Khattab & Zaharia, 2020) keeps one vector per token instead, and compares a query and a passage by letting each query token find its best match.

Let $\mathbf{Q} \in \mathbb{R}^{n_q \times d}$ hold the normalized token embeddings of the query (row $\mathbf{q}_i$) and $\mathbf{P} \in \mathbb{R}^{n_p \times d}$ those of the passage (row $\mathbf{p}_j$). The score is

In plain words, for each query token, find the most similar passage token, and add up those best matches. This is called MaxSim, and the design is called late interaction, because query and passage are encoded independently (like a bi-encoder, so passages can be precomputed) but interact at the token level at the very end (a little like a cross-encoder).

ColBERT's details are worth knowing. A linear layer projects BERT's 768-dimensional token states down to $d = 128$. Queries are padded with [MASK] tokens to a fixed length of 32, which the authors call query augmentation, and the masked positions learn to act like soft expansion terms. Special [Q] and [D] markers tell the shared encoder which side it's encoding. The paper reports quality competitive with BERT cross-encoder rerankers while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query.

Here's MaxSim by hand, with a two-token query and a three-token passage in two dimensions.

Check it by hand. The first query token's best match is the third passage token (cosine 0.96). The second query token's best match is the first passage token (also 0.96), not the second (0.80). So the score is $0.96 + 0.96 = 1.92$. Notice that a passage token can be the best match for several query tokens, or for none, and that each query token contributes on its own, so a passage that nails one term but misses another scores lower than one that covers both.

The price is storage. One vector per token means a 100-token passage stores 100 vectors of 128 dimensions, which is 25.6 KB at 16 bits, against 1.5 KB for one 768-dimensional float16 vector. ColBERTv2 (Santhanam et al., 2022) attacks this with residual compression, storing each token vector as the id of its nearest centroid plus a 1 or 2 bit quantized residual, which cuts the footprint by 6 to 10 times while also improving quality through distillation from a cross-encoder. Search runs in two steps, first finding candidate passages through their token vectors' centroids, then scoring the candidates with full MaxSim.

import numpy as np

def maxsim(Qt, q_mask, Pt, p_mask):
    raise NotImplementedError

MTEB, and how to read a leaderboard

How do you pick among hundreds of models? The Massive Text Embedding Benchmark (MTEB, Muennighoff et al., 2022) was built to stop people from evaluating on one dataset of one task. The original version covers 8 task types, 58 datasets, and 112 languages, and its headline finding is still true. No single model wins every task.

The retrieval tasks come from BEIR (Thakur et al., 2021), 18 datasets from different domains, designed to test zero-shot transfer. Its original finding is a useful prior. BM25 (§7.5) is a robust baseline that many dense models of the time failed to beat out of domain. MTEB has since grown into a massive multilingual edition, MMTEB (Enevoldsen et al., 2025), with hundreds of tasks.

Treat the leaderboard as a shortlist, not an answer.

• Look at the task column you care about, not the average. A model that's best on classification can be mediocre at retrieval.
• Check for contamination. Many models train on the training splits of datasets that MTEB tests on, and some on data that overlaps test sets. A model that's suspiciously good on a few datasets may have seen them.
• Read the model card's prefixes and maximum length. Benchmarks are run with the right instructions, and your pipeline must use them too.
• Weigh cost. Dimension, parameter count, and maximum sequence length decide your index size and latency.
• Evaluate on your own data. A few hundred real queries with judged results, scored with recall@k and nDCG@10, tell you more than any public average. §7.6 shows how to build such a set.

In PyTorch

The Matryoshka loss is a loop over prefix sizes inside an ordinary InfoNCE, and bit packing for binary codes takes a few tensor operations.

With the sentence-transformers library, prefixes, truncation, and quantization are all options. This needs a model download, so it's shown and not run.

Read more

• Wang et al. (2022), Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5), and Izacard et al. (2022), Unsupervised Dense Information Retrieval with Contrastive Learning (Contriever). Weakly supervised pairs with consistency filtering, prefixes, and the batch-size ablation, and contrastive pretraining with no labels at all.
• Li et al. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning (GTE). Data mixtures, the enlarged-negative loss, and scaling studies.
• Xiao et al. (2023). C-Pack: Packed Resources For General Chinese Embeddings (BGE). The three-stage BGE recipe with instructions.
• Wang et al. (2024). Improving Text Embeddings with Large Language Models (E5-Mistral). Synthetic data, last-token pooling, and the instruction ablation.
• Kusupati et al. (2022). Matryoshka Representation Learning. Nested losses and adaptive retrieval.
• Shakir, Aarsen & Lee (2024). Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval. A practical study with rescoring.
• Khattab & Zaharia (2020), ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, and Santhanam et al. (2022), ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. Late interaction and residual compression.
• Muennighoff et al. (2022), MTEB: Massive Text Embedding Benchmark, and Thakur et al. (2021), BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. The benchmarks, and why BM25 is a strong zero-shot baseline.

Watch next

▶

▶

▶

▶