"Dog bites man" and "man bites dog" contain the same three tokens. To a Transformer built from the parts we have so far, they are the same sentence.

That claim should bother you, so let's check it. In §4.6 we saw that self-attention without a mask is permutation equivariant. Shuffle the input tokens and the outputs come out shuffled the same way, with identical values. The FFN processes each token alone, and LayerNorm normalizes each token alone, so they are equivariant too. A stack of such layers can tell you which tokens are present and how they relate by content, but it has no idea of their order. For language, where order carries most of the meaning, that is fatal.

The first line prints True. Without a mask, order is invisible. The second prints False, which is a hint we will come back to at the end. A causal mask is itself a statement about order, since token $t$ may only look at tokens up to $t$.

There are two broad ways to give a model a sense of position. We can tag each token's input vector with its absolute position before the first layer, or we can change attention itself so that its scores depend on the relative offset between query and key. This section covers the first family and the classic relative schemes. Section 5.5 covers the two that dominate modern models, RoPE and ALiBi.

Sinusoidal encodings

The original Transformer (Vaswani et al., 2017) adds a fixed vector $\mathbf{p}_t \in \R^d$ to the embedding $\mathbf{e}_t \in \R^d$ of the token at position $t$, so the first layer sees $\mathbf{e}_t + \mathbf{p}_t$. (In this section $\mathbf{e}_t$ is that embedding row, not a one-hot vector.) The vector is built from sines and cosines at $d/2$ geometrically spaced frequencies,

Each pair of coordinates $(2i, 2i+1)$ is a point going around a circle as $t$ increases, at angular speed $\omega_i$. The first pair turns once every $2\pi \approx 6.3$ tokens. The last turns once every $2\pi \cdot 10000^{(d-2)/d}$ tokens, about 60,600 for $d = 512$ (and approaching $2\pi\cdot 10000 \approx 62{,}800$ as $d$ grows). The wavelengths in between grow geometrically.

In plain words, a position is written as the readings of $d/2$ clocks that tick at very different speeds, from a second hand to a hand that takes tens of thousands of steps to go around.

A binary counter is a good mental picture. Its lowest bit flips every step, the next every two steps, the next every four, and together the bits name every integer uniquely. The sinusoidal encoding is a smooth version, with continuous values instead of bits and a geometric ratio of $10000^{2/d}$ instead of 2 between neighboring frequencies.

A worked example with $d = 4$. The frequencies are $\omega_0 = 1$ and $\omega_1 = 10000^{-1/2} = 0.01$. Position 0 is $(0, 1, 0, 1)$, since every sine is 0 and every cosine is 1. Position 1 is $(\sin 1, \cos 1, \sin 0.01, \cos 0.01) \approx (0.841, 0.540, 0.010, 1.000)$. Position 2 is $(0.909, -0.416, 0.020, 1.000)$. The fast pair has already swung far, while the slow pair has barely moved.

Vaswani et al. also multiplied the token embeddings by $\sqrt d$ before adding the encoding. The paper does not say why. The usual explanation is scale. Embeddings initialized with entries of size about $1/\sqrt d$ would otherwise be swamped by position vectors whose entries are of order one.

Property 1, a shift is a rotation

Why sines and cosines, rather than any other set of distinct vectors? Vaswani et al. hypothesized that this choice makes relative positions easy to compute, because the encoding of position $t + k$ is a fixed linear function of the encoding of position $t$. Use the angle-addition formulas on one pair,

Stack one such $2\times2$ rotation for every frequency into a block-diagonal matrix $\mathbf{M}_k \in \R^{d\times d}$, and

In plain words, moving $k$ tokens forward rotates each clock by $k$ ticks of its own speed, and that rotation does not care where you started.

This is a theorem about the encodings, and it is the seed of RoPE in §5.5. It does not by itself make the model use relative positions, because the model never sees $\mathbf{p}_t$ alone (more on that below).

The derivation also answers a question people often skip, which is why each frequency gets both a sine and a cosine. A sine alone fails twice. It does not pin down the angle, since $\sin a = \sin(\pi - a)$, so one sine reading cannot tell an angle from its mirror image. And the shift is no longer linear, because $\sin((t+k)\omega)$ needs $\cos(t\omega)$, which a sine-only code does not contain. The pair puts each frequency on a circle, and on a circle a shift is a rotation.

import numpy as np

def sinusoidal_pe(T, d, base=10000.0):
    raise NotImplementedError

Property 2, dot products depend only on the offset

Take two positions $t$ and $s$. Each pair contributes $\sin t\omega_i\sin s\omega_i + \cos t\omega_i\cos s\omega_i = \cos\big((t-s)\omega_i\big)$, so

In plain words, how similar two position vectors are depends only on how far apart they are, not on where they sit in the sequence.

The same holds for distances. Expanding the square and using $\lVert\mathbf{p}_t\rVert^2 = d/2$ for every $t$,

so every pair of neighbors is the same distance apart, wherever they sit. For $d = 512$ that neighbor distance is $\sqrt{13.8} \approx 3.7$, most of it from the fastest few dozen clocks, whose angles change most per step.

Two more facts follow from the similarity formula. It is symmetric, $\mathbf{p}_t\cdot\mathbf{p}_{t+k} = \mathbf{p}_t\cdot\mathbf{p}_{t-k}$, so the dot product alone cannot tell "before" from "after" (the rotation $\mathbf{M}_k$ can, since $\mathbf{M}_{-k} = \mathbf{M}_k^\top \ne \mathbf{M}_k$). And it tends to fall as $|t - s|$ grows, because the fast clocks dephase quickly and only the slow ones stay aligned. The fall is not monotonic, but large offsets look much less alike than small ones.

Choosing the base

The base $b = 10{,}000$ sets the range of wavelengths, from $2\pi$ tokens for the fastest pair to about $2\pi b$ for the slowest. The two ends do different jobs. The fast clocks tell neighbors apart, and the slow clocks say roughly how far along the sequence a token is, smoothly, so that offsets of 64 and of 1,000 look different. If the longest wavelength is shorter than the context, every slow clock wraps around within one sequence and that smooth coordinate is gone. The paper does not explain its choice, but a base of 10,000 puts the longest wavelength at tens of thousands of tokens, far beyond the sentences of the original translation task. RoPE inherits the same frequencies, and long-context models raise the base for exactly this reason (LLaMA 3 uses 500,000, §5.5).

With base 100 the longest wavelength is about 540 tokens, and offsets of 64 and 1,000 are equally unrelated to position 0 (similarity near 0). With base 10,000 similarity falls gradually, 0.43 at offset 64 and 0.28 at 1,000, so distance remains readable across the whole context. A larger base buys smoothness over longer ranges at the price of slightly less contrast between near neighbors.

How the model actually sees position

Here is the catch that most explanations skip. Attention never computes $\mathbf{p}_m\cdot\mathbf{p}_n$. It computes a query from the sum $\mathbf{e}_m + \mathbf{p}_m$ and a key from $\mathbf{e}_n + \mathbf{p}_n$, so the score between query position $m$ and key position $n$ expands into four terms,

with $\mathbf{W} = \mathbf{W}_Q\mathbf{W}_K^\top \in \R^{d\times d}$. Write $\mathbf{p}_m = \mathbf{M}_m\mathbf{p}_0$ (as columns) and the position term becomes $\mathbf{p}_0^\top\mathbf{M}_m^\top\mathbf{W}\mathbf{M}_n\mathbf{p}_0$. It is guaranteed to depend on $m - n$ alone when $\mathbf{W}$ commutes with every $\mathbf{M}_k$ (for example, if it is itself block-diagonal with $2\times2$ blocks that are scaled rotations), because then $\mathbf{M}_m^\top\mathbf{W}\mathbf{M}_n = \mathbf{W}\mathbf{M}_{n-m}$. Nothing forces that. The model can learn relative attention from sinusoids, but it is not built in, and the mixed terms let content and absolute position interfere. Ke et al. (2021) argued that these mixed terms mostly add noise and proposed computing content and position scores with separate projections. The rest of this section, and all of §5.5, are attempts to build relative position into the score directly.

import numpy as np

def shift_matrix(k, d, base=10000.0):
    raise NotImplementedError

With $\mathbf{M}_k$ in hand, we can show that a head can learn exact relative attention from sinusoids, even though nothing forces it to. The construction is a classic interview question. Choose $\mathbf{W} = c\,\mathbf{M}_k$ for a gain $c > 0$. Then, writing the position vectors as rows, $\mathbf{p}_m\mathbf{W}\mathbf{p}_n^\top = c\,(\mathbf{M}_k^\top\mathbf{p}_m^\top)\cdot\mathbf{p}_n = c\,\mathbf{p}_{m-k}\cdot\mathbf{p}_n$, because $\mathbf{M}_k^\top = \mathbf{M}_{-k}$. By Property 2 this depends only on $m - n$ and peaks at $n = m - k$, where it equals $c\,d/2$. So a head whose query and key projections multiply to $c\,\mathbf{M}_k$ on the position subspace attends $k$ tokens back, at every position. With $k = 1$ it is the previous-token head that an induction circuit starts from (§4.7).

import numpy as np

def relative_head_W(k, d, c=8.0, base=10000.0):
    raise NotImplementedError

The same clocks, beyond text

Sines and cosines at geometric frequencies are a general way to feed a scalar to a network so that small and large differences are both visible. Diffusion models embed the noise level, or timestep, of their input exactly this way before adding it to every block, so that a diffusion model can tell nearly clean images from nearly pure noise. Coordinate networks do the same with positions in space. Tancik et al. (2020) showed that passing raw coordinates through such Fourier features is what lets a small MLP fit fine detail, and NeRF's positional encoding (Mildenhall et al., 2020) is one instance. In every case the fast frequencies resolve fine detail and the slow ones give a smooth sense of scale.

Learned absolute positions

The simplest alternative is to learn the position vectors. Keep a table $\mathbf{P} \in \R^{T_{\max}\times d}$, exactly like the token embedding table of §1.2, and add row $t$ to the token at position $t$. BERT uses $T_{\max} = 512$ and GPT-2 uses $T_{\max} = 1024$, which costs $1024 \times 768 = 786{,}432$ parameters in GPT-2 small. Vaswani et al. tried this too and reported nearly identical results to the sinusoids on translation.

Learned positions are flexible, since the model can put whatever structure it finds useful into the table. Their limitation is absolute. There is no row for position $T_{\max}$, so the model cannot even be run on a longer input, let alone generalize to one. Positions near the end of the table also get fewer useful gradient updates if most training documents are shorter than $T_{\max}$.

Relative positions

For most of language, what matters is not that a word is the 517th token of a document but that it is two tokens after the verb. That suggests building position into attention as a function of the offset $n - m$ between the key position $n$ and the query position $m$. A relative scheme is translation invariant by construction (the same pattern at any position gets the same scores), and it does not need a table indexed by absolute position.

Shaw et al., relative vectors on keys and values

Shaw, Uszkoreit & Vaswani (2018) learned a vector $\mathbf{a}_r \in \R^{d_k}$ for each clipped offset $r = \operatorname{clip}(n - m, -K, K)$ and added it inside attention,

where $\mathbf{q}_m, \mathbf{k}_n, \mathbf{v}_n \in \R^{d_k}$ are a head's query, key, and value, $\alpha_{mn}$ the softmax weights, and $K$ the clipping distance. Offsets farther than $K$ share a vector, which caps the parameters at $2K + 1$ vectors per table. On WMT 2014 translation this gained 1.3 BLEU over absolute positions on English to German and 0.3 on English to French. Varying $K$ made little difference once $K \ge 2$, because stacked layers can propagate precise position information beyond the clipping window.

The naive implementation materializes a $(T, T, d_k)$ tensor of relative vectors, which is expensive. The efficient trick is to compute $\mathbf{q}_m\cdot\mathbf{a}_r$ for every query and every one of the $2K+1$ offsets once, a $(T, 2K+1)$ matrix, and gather from it. Huang et al. (2018) found a "skewing" reshape that does the same for unclipped offsets with $O(Td)$ extra memory.

T5, one learned scalar per offset bucket

Raffel et al. (2020) simplified further for T5. Each head adds a single learned scalar to each attention logit, chosen by the offset,

where $b_i$ is the bias table of head $i$ (one of the layer's $h$ heads). Each $b_i$ has 32 entries, and the tables are shared by all layers of a stack. Offsets are grouped into buckets, exact for small distances and logarithmically wider for large ones, up to 128 tokens, beyond which all offsets share a bucket. For an encoder (bidirectional), half the buckets are for keys before the query and half for keys after. Within each half, the first 8 buckets hold the distances 0 to 7 exactly, then the buckets cover distances 8 to 11, 12 to 15, 16 to 22, 23 to 31, 32 to 45, 46 to 63, 64 to 90, and finally everything from 91 on. A causal decoder only needs keys at or before the query, so it uses all 32 buckets for that direction, with 16 exact and 16 logarithmic.

In plain words, the model learns how much each head likes to look one token back, two tokens back, and so on, with coarser resolution for faraway tokens, whose precise distance rarely matters.

The design is cheap in parameters ($32h$ per stack, so the encoder and the decoder each have their own), and because all far offsets share one bucket, nothing in the scheme changes when the input gets longer than anything seen in training. Press et al. (2022) found that T5's bias extrapolates to longer inputs better than sinusoidal or rotary encodings do, but is considerably slower, since the bias must be materialized for every query-key pair.

import numpy as np

def t5_bucket(rel, bidirectional=True, num_buckets=32, max_distance=128):
    raise NotImplementedError

import numpy as np

def relative_bias_attention(Q, K, V, bias_table):
    raise NotImplementedError

Length generalization

Every position scheme is ultimately judged by one question. Train on sequences of length $T$, then evaluate on longer ones. Does the model still work? This is called length generalization or extrapolation, and it matters because training cost grows with sequence length, so you would like to train short and deploy long.

Each scheme fails or succeeds for a reason you can now state. Learned absolute positions cannot run at all beyond $T_{\max}$. Sinusoidal encodings can be computed for any $t$, but the model has never seen the combinations of slow-clock angles that occur beyond its training length. Press et al. (2022) found that a sinusoidal model trained on 1,024 tokens improved only up to about 50 extra tokens and then degraded quickly. Relative schemes with clipping or bucketing (Shaw, T5) see nothing new at longer lengths, because every offset beyond the last bucket looks the same, and they extrapolate better. The cost is that the model is blind to distinctions between large offsets.

There is a fourth option that sounds absurd and is not. A causal decoder with no position encoding at all, called NoPE, can still learn about position, because the causal mask makes the computation order-dependent. Token $t$ attends over exactly $t$ tokens, so, for example, a head that spreads its attention evenly gives each visible token weight $1/t$, and its output carries information about $t$. Haviv et al. (2022) showed that causal language models without positional encodings are competitive with ones that have them, and conjectured exactly this mechanism, counting predecessors. Kazemnejad et al. (2023) found that NoPE generalized to longer lengths better than common explicit schemes on their reasoning and math tasks. These are empirical results at limited scale. At the frontier, the dominant recipe is RoPE plus a dedicated stage that extends the context, which we preview in §5.5 and develop in §12.7.

In PyTorch

The sinusoidal table is a buffer, not a parameter. It moves with the model between devices and receives no gradient, and registering it with persistent=False keeps it out of the state dict, since it can always be recomputed. Learned positions are an nn.Embedding.

Read more

• Vaswani et al. (2017). Attention Is All You Need. Section 3.5 introduces the sinusoidal encoding and its shift hypothesis.
• Shaw, Uszkoreit & Vaswani (2018). Self-Attention with Relative Position Representations. Relative vectors inside attention, with the clipping ablation.
• Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. T5, including its bucketed relative bias (Section 2.1).
• Huang et al. (2018). Music Transformer. The skewing trick for memory-efficient relative attention.
• Ke, He & Liu (2021). Rethinking Positional Encoding in Language Pre-training. The four-term expansion and why mixing content with position hurts.
• Haviv, Ram, Press, Izsak & Levy (2022). Transformer Language Models without Positional Encodings Still Learn Positional Information. NoPE.
• Kazemnejad et al. (2023). The Impact of Positional Encoding on Length Generalization in Transformers. A careful comparison of position schemes for extrapolation.
• Tancik et al. (2020). Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Why sinusoidal features of a coordinate help an MLP, beyond language.

Watch next

▶