Section 5.4 ended with a wish. We want an attention score between a query at position $m$ and a key at position $n$ that depends on their positions only through the offset $m - n$, exactly and in every layer. We would like it to cost no parameters and no $T \times T$ table, and we do not want to lose any information about content along the way.

There is a beautiful way to get all of that, and you have already seen its key ingredient. In §5.4 the shift from position $t$ to $t + k$ turned out to be a rotation. Rotations have a property that makes them perfect for the job. If you rotate two vectors by different angles, their dot product depends only on the difference of the angles. So instead of adding position to the token, rotate each query and key by an angle proportional to its position. That is rotary position embedding, RoPE (Su et al., 2021), and it is the position scheme of LLaMA, Mistral, Qwen, Gemma, DeepSeek, and most other open models.

This section derives RoPE from complex numbers, proves its relative-position property, implements it the two ways real codebases do, and then covers ALiBi (Press, Smith & Lewis, 2022), which takes the opposite approach and directly penalizes distance.

Two dimensions first, with complex numbers

Start with a single pair of coordinates. A vector $(q_0, q_1) \in \R^2$ is the same thing as the complex number $\tilde q = q_0 + \mathrm{i}\,q_1$, where $\mathrm{i}$ is the imaginary unit. The real dot product of two such vectors is the real part of one times the conjugate of the other,

Multiplying by $e^{\mathrm{i}\alpha} = \cos\alpha + \mathrm{i}\sin\alpha$ rotates a complex number by the angle $\alpha$ without changing its length. Now rotate the query at position $m$ by $m\omega$ and the key at position $n$ by $n\omega$, for some fixed frequency $\omega$. Their score is

because $\overline{e^{\mathrm{i}n\omega}} = e^{-\mathrm{i}n\omega}$ and the exponents add.

In plain words, the score of a rotated query and a rotated key equals the score of the original pair with the key rotated by the difference of their angles. Absolute positions cancel, and only $m - n$ survives.

A worked example. Take $\mathbf{q} = \mathbf{k} = (1, 0)$ and $\omega = 0.5$. At $m = 3$ the query is rotated by 1.5 radians to $(\cos 1.5, \sin 1.5) \approx (0.071, 0.997)$. At $n = 1$ the key is rotated by 0.5 radians to $(0.878, 0.479)$. Their dot product is $0.071 \cdot 0.878 + 0.997 \cdot 0.479 \approx 0.540$, which is $\cos(1.0)$, the cosine of $(3 - 1)\cdot 0.5$. Move both to $m = 10$ and $n = 8$ and you get exactly the same number.

Deriving RoPE from what we want

Su et al. did not guess rotations. They wrote down the property they wanted and solved for the encoding. The derivation is short and worth knowing, because it shows that rotation is not just one option. Under mild assumptions, it is the solution.

Any dimension, block-diagonal rotations

A head's query and key live in $\R^{d_k}$, not $\R^2$. RoPE splits them into $d_k/2$ pairs and rotates pair $j$ at its own frequency, reusing the geometric frequencies of the sinusoidal encoding,

where $b$ is the base (10,000 in the original paper). With column vectors and the standard counterclockwise rotation $\mathbf{R}(\alpha) = \begin{pmatrix}\cos\alpha & -\sin\alpha\\ \sin\alpha & \cos\alpha\end{pmatrix}$, the rotation for position $m$ is the block-diagonal matrix

and RoPE replaces each query and key by its rotated version, $\mathbf{q}_m \mapsto \mathbf{R}_m\mathbf{q}_m$ and $\mathbf{k}_n \mapsto \mathbf{R}_n\mathbf{k}_n$, after the projections and before the dot product. The values are not rotated.

The relative-position property, proved. Every block is a rotation, so $\mathbf{R}(\alpha)^\top = \mathbf{R}(-\alpha)$, and rotations of the same plane compose by adding angles, $\mathbf{R}(\alpha)\mathbf{R}(\beta) = \mathbf{R}(\alpha + \beta)$. Block-diagonal matrices multiply block by block, so $\mathbf{R}_m^\top\mathbf{R}_n = \mathbf{R}_{n-m}$. Therefore

In plain words, the score between a query at $m$ and a key at $n$ is the score of the unrotated query against the key rotated by the offset. It depends on content through $\mathbf{q}$ and $\mathbf{k}$, and on position only through $n - m$. This is exact, at every layer, for any learned $\mathbf{W}_Q$ and $\mathbf{W}_K$, which is precisely what added sinusoids could not guarantee.

Several other properties fall out immediately.

• No parameters and no change of scale. Rotations preserve the norms of $\mathbf{q}$ and $\mathbf{k}$, so RoPE never changes how large a logit can be (it stays within $\pm\lVert\mathbf{q}\rVert\lVert\mathbf{k}\rVert/\sqrt{d_k}$), only how it depends on position.
• Position enters every layer. RoPE is applied inside every attention layer, so every layer sees exact relative positions, instead of whatever trace of an input-layer encoding has survived in the stream.
• Values stay position-free. Since $\mathbf{v}$ is not rotated, position affects which tokens are mixed but not what is written into the stream.
• It works with a KV cache. A key's rotation depends only on its own position, so keys can be rotated once, cached, and reused (§6.5).

A gentle decay with distance

Su et al. also showed that, with the geometric frequencies, the largest possible score between a query and a key tends to shrink as their distance grows. Here is the idea in a picture you can compute. Take a query and a key with identical content, and give every pair the same magnitude. Then the score at offset $\Delta$ is $\sum_j\cos(\Delta\,\omega_j)$, which is exactly the sinusoidal similarity curve of §5.4. It starts at its maximum $d_k/2$ and falls off, with ripples, as the fast pairs dephase.

Implementing it without matrices, and the two conventions

Nobody builds $\mathbf{R}_m$. For each pair, the rotation is two multiplies and an add, which vectorizes as

where $\boldsymbol{\omega}'$ repeats each frequency twice to line up with its pair, and $\operatorname{rot}(\mathbf{x})$ maps each pair $(x_a, x_b)$ to $(-x_b, x_a)$. The cosines and sines depend only on positions, so they are computed once for the maximum length and cached.

There are two conventions for which coordinates form a pair, and this trips up many people.

Interleaved, as in the RoFormer paper, GPT-J, and Meta's original LLaMA code, pairs adjacent coordinates $(x_{2j}, x_{2j+1})$. Half-split, as in GPT-NeoX and the Hugging Face LLaMA implementation, pairs $x_j$ with $x_{j + d_k/2}$, so $\operatorname{rot}(\mathbf{x}) = (-\mathbf{x}_{d_k/2:},\ \mathbf{x}_{:d_k/2})$, often written rotate_half. The two are the same function up to a fixed permutation of the $d_k$ coordinates, and a permutation of the coordinates of $\mathbf{q}$ and $\mathbf{k}$ can be absorbed into $\mathbf{W}_Q$ and $\mathbf{W}_K$ (their columns in this book's $\mathbf{x}\mathbf{W}$ convention, the rows of a PyTorch nn.Linear weight). Applying the same permutation to both leaves every dot product unchanged. They are equally good for training. But a checkpoint trained with one convention gives garbage under the other, which is why the script that converts Meta's LLaMA weights to the Hugging Face format permutes the query and key projection weights.

import numpy as np

def rope_cos_sin(positions, d, base=10000.0):
    raise NotImplementedError

import numpy as np

def apply_rope(x, positions, base=10000.0):
    raise NotImplementedError

import numpy as np

def interleaved_to_half_perm(d):
    raise NotImplementedError

Where RoPE goes in a real attention layer

In a multi-head layer, RoPE acts on each head separately, after the projections and the split into heads, with frequencies computed from the head dimension $d_k$, not the model width $d$. A LLaMA attention layer computes $\mathbf{Q}, \mathbf{K}, \mathbf{V}$, reshapes them to $(h, T, d_k)$ for $h$ heads (keys and values to $(n_{kv}, T, d_k)$ with grouped-query attention), rotates $\mathbf{Q}$ and $\mathbf{K}$, and then runs ordinary causal attention. Some models rotate only part of each head. GPT-NeoX-20B rotates a quarter of each head's dimensions and leaves the rest position-free, and DeepSeek's multi-head latent attention keeps a separate small rotated component (§12.5).

import numpy as np

def rope_attention(X, Wq, Wk, Wv, Wo, H, offset=0):
    raise NotImplementedError

The base, and a preview of context extension

The base $b$ sets the slowest rotation. Pair $j$ completes a full turn every $2\pi b^{2j/d_k}$ tokens, so with $d_k = 128$ and $b = 10{,}000$ the slowest pair has a wavelength of about 54,000 tokens, while the fastest turns every 6.3 tokens. LLaMA 3 raised the base to 500,000, which pushes the slowest wavelength past 2.5 million tokens.

Why would that matter? Consider a model trained on sequences of 4,096 tokens. Its fast pairs have rotated through many full circles during training and have seen every angle. Its slow pairs have only ever seen a small arc. Run the model on 32,000 tokens and the slow pairs reach angles it never saw, and attention degrades. Press et al. (2022) found that rotary models extrapolate a little better than sinusoidal ones (on the order of 100 to 200 extra tokens in their setups) but nowhere near the lengths people want.

The standard fixes all manipulate the angles. Position interpolation (Chen et al., 2023) divides every position by the extension factor, so 32,000 tokens are squeezed into the angle range of 4,096, and a short fine-tune (under 1,000 steps in their experiments) adapts the model. Raising the base (sometimes called NTK-aware scaling) slows the low frequencies while barely touching the high ones. YaRN (Peng et al., 2023) interpolates different frequency bands by different amounts and adjusts the attention temperature. All of this is the subject of §12.7. For now, notice that every one of these methods is possible only because RoPE makes position an explicit, adjustable angle.

ALiBi, a penalty for distance

Press, Smith & Lewis (2022) took the most direct route imaginable. Add no position information to the tokens at all, and subtract from each attention logit a penalty proportional to the distance,

for head $i$, where $\lambda_i > 0$ is a fixed, head-specific slope (we write $\lambda_i$ for what the paper calls $m$, since $m$ is our query position). For $h$ heads the slopes are a geometric sequence,

so 8 heads get $\tfrac12, \tfrac14, \dots, \tfrac1{256}$. The slopes are not learned.

In plain words, every head starts with a built-in preference for recent tokens, and each head has a different horizon.

Put next to §5.4, ALiBi is T5's relative bias with the learned table replaced by a fixed straight line, one slope per head, and no buckets, so it costs no parameters and nothing changes at lengths never seen in training.

The horizon has a precise meaning. Softmax ignores any constant added to a whole row of logits, and $-\lambda_i(m - n) = -\lambda_i m + \lambda_i n$, where the first term is constant along the row. So ALiBi multiplies the attention weight of a key $\Delta$ tokens back by $e^{-\lambda_i\Delta}$ relative to the current token. That is an exponential recency prior with a decay length of $1/\lambda_i$ tokens. The steepest of 8 heads ($\lambda = 1/2$) mostly sees the last few tokens, and the gentlest ($\lambda = 1/256$) can look back hundreds.

ALiBi was designed for extrapolation, and it delivers. In the paper, a 1.3-billion-parameter model trained on 1,024 tokens and evaluated on 2,048 matched the perplexity of a sinusoidal model trained on 2,048, while training 11% faster and using 11% less memory. BLOOM and MPT used it. The catch is the flip side of the prior. Distant tokens are penalized by construction, and the paper's own analysis is candid about what that means. Its extrapolation gains come largely from avoiding the "early token curse" (in non-overlapping evaluation, the first tokens of each window have almost no context), and with sliding-window evaluation the ALiBi models did not improve perplexity by looking further back than their training length. Good perplexity on long inputs is not the same as using a long context well. In practice, models that must retrieve facts from far away have standardized on RoPE with context extension.

For a head count that is not a power of 2, the authors' code takes the slopes for the largest power of 2 below $h$, then fills the remaining heads with every other slope from the sequence for twice that power.

import numpy as np, math

def alibi_slopes(H):
    raise NotImplementedError

def alibi_bias(T, H):
    raise NotImplementedError

In PyTorch

The half-split convention is the common one in PyTorch codebases, because rotate_half is a cheap concatenation. ALiBi plugs into F.scaled_dot_product_attention as an additive mask.

Every position scheme, compared

Read more

• Su et al. (2021). RoFormer, Enhanced Transformer with Rotary Position Embedding. The derivation, the long-term decay bound, and RoPE with linear attention.
• Press, Smith & Lewis (2022). Train Short, Test Long, Attention with Linear Biases Enables Input Length Extrapolation. ALiBi, with careful extrapolation comparisons of every scheme.
• Chen et al. (2023). Extending Context Window of Large Language Models via Positional Interpolation. The simplest RoPE context extension and why extrapolation fails.
• Peng et al. (2023). YaRN, Efficient Context Window Extension of Large Language Models. Frequency-dependent interpolation.
• EleutherAI (2021). Rotary Embeddings, A Relative Revolution. A clear explanation with implementation notes and early experiments.
• Kazemnejad et al. (2023). The Impact of Positional Encoding on Length Generalization in Transformers. How RoPE, ALiBi, T5, and NoPE compare on reasoning tasks.

Watch next

▶

▶

▶