In 2014 Dzmitry Bahdanau, working with Kyunghyun Cho and Yoshua Bengio, looked at the encoder-decoder of §3.5 and asked a simple question. Why should the decoder get only one vector? The encoder already computes a state for every source word. Throwing all of them away except the last is like asking a translator to read a sentence once, close the book, and translate from memory.

Their fix, published as Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau, Cho, and Bengio 2015), lets the decoder look back. Before writing each target word, it scores every source position for relevance, turns the scores into weights that sum to one, and reads a weighted average of the encoder states. Each target word gets its own context vector, built on demand. The paper called it an alignment model. Everyone else came to call it attention, and within three years it had eaten the recurrence it was added to. This section builds Bahdanau's attention, then Luong's simpler variants, looks at the alignments they learn, and follows the short path from "RNN plus attention" to "attention is all you need".

Let the decoder look at every encoder state

Start with the encoder. Bahdanau et al. used a bidirectional RNN, exactly the forward and backward pair of §3.4, so that the state for source word $j$ summarizes the words around it on both sides. They called each concatenated state an annotation,

Now the decoder. Suppose it has state $\mathbf s_{i-1} \in \R^{d_s}$ after writing $i-1$ target words, and is about to write word $i$. Attention runs in three steps.

1. Score. For every source position $j$, compute how relevant annotation $\mathbf h_j$ is to the decoder's current state,

2. Normalize. Turn the scores into weights over the source with a softmax,

3. Read. The context for this step is the weighted average of the annotations,

The decoder then updates its state with the previous word and this context, $\mathbf s_i = f(\mathbf s_{i-1}, y_{i-1}, \mathbf c_i)$, and predicts the next word from all three, $p(y_i \mid y_{<i}, \mathbf x) = g(y_{i-1}, \mathbf s_i, \mathbf c_i)$, where $f$ is a gated RNN cell and $g$ an output layer with a softmax.

In plain words, at every step the decoder asks "which source words matter for what I am about to write?", answers with a probability distribution over source positions, and reads the source through that distribution. The score function is a one-hidden-layer network that takes the decoder state and one annotation and outputs a number. Because it adds a projection of each input inside the tanh, it is called additive attention (and also concat or MLP attention). Nothing about it is fixed by hand. $\mathbf W_a$, $\mathbf U_a$, and $\mathbf v_a$ are trained with everything else, by backpropagating the translation loss through the softmax.

A small worked example, which you can check by hand. Take $d_s = 2d_h = n' = 2$, $\mathbf W_a = \mathbf U_a = \mathbf I$, $\mathbf v_a = (1, 1)$, decoder state $\mathbf s = (1, 0)$, and three annotations $\mathbf h_1 = (1, 0)$, $\mathbf h_2 = (0, 1)$, $\mathbf h_3 = (-1, 0)$. Then $\mathbf s + \mathbf h_1 = (2, 0)$, so $e_1 = \tanh 2 + \tanh 0 = 0.964$. Next $\mathbf s + \mathbf h_2 = (1, 1)$, so $e_2 = 2\tanh 1 = 1.523$, and $\mathbf s + \mathbf h_3 = (0, 0)$, so $e_3 = 0$. The softmax gives $\boldsymbol\alpha = (0.319, 0.559, 0.122)$, and the context is $\mathbf c = 0.319\,\mathbf h_1 + 0.559\,\mathbf h_2 + 0.122\,\mathbf h_3 = (0.198, 0.559)$. Notice that $\mathbf h_2$ won even though $\mathbf h_1$ points the same way as $\mathbf s$. An additive score is not a similarity. It is whatever function the trained weights make it.

import numpy as np

def additive_attention(s, H, Wa, Ua, va, mask=None):
    e = np.einsum("bd,btd->bt", s, H)                 # a dot product is not additive attention
    a = np.exp(e) / np.exp(e).sum(1, keepdims=True)
    return np.einsum("bt,btd->bd", a, H), a

One implementation detail matters for speed. The term $\mathbf U_a\mathbf h_j$ does not depend on the decoder step $i$, so it is computed once per sentence for all $j$ and reused at every step. Only $\mathbf W_a\mathbf s_{i-1}$ changes. That is why the hint above isolates H @ Ua, and why the PyTorch model below precomputes UH.

Attention fixes the bottleneck

Compare what the decoder can see with and without attention. Without it, all source information arrives through one vector, and a fact about the first source word must survive the whole encoder and every decoder step before it is used. With attention, every decoder step reads directly from every annotation. The path from any source word to any target word is one attention hop, whatever the sentence length. Three things follow.

1. Capacity grows with the input. The decoder has access to $T_x$ vectors instead of one, the "keep all $n$ vectors" line in the widget of §3.5. Nothing has to be compressed in advance.
2. Gradients take a shortcut. The loss at target position $i$ sends gradient straight to every $\mathbf h_j$ through $\alpha_{ij}$, without crossing a long chain of recurrent Jacobians. Attention is as much a fix for the vanishing gradients of §3.3 as for the bottleneck.
3. The model reads what it needs, when it needs it. Each target word gets its own context, so the encoder no longer has to guess in advance which facts the decoder will want.

The results were decisive. On WMT'14 English to French, with models trained on sentences of up to 50 words, Bahdanau et al. report the following BLEU scores (Table 1 of the paper).

Attention improved the encoder-decoder by about 9 BLEU on all sentences, and on sentences without unknown words the longer-trained attention model passed the phrase-based system. Their length plot told the same story more vividly. The encoder-decoder's quality collapsed as sentences got longer, while the attention model, in the paper's words, "shows no performance deterioration even with sentences of length 50 or more". (The remaining gap on all sentences came largely from unknown words, since the models had a 30,000-word vocabulary, a problem that subword tokenization from Chapter 2 would soon solve.)

We can reproduce the effect in miniature. Here is the digit-reversal model of §3.5, now with a bidirectional encoder and Bahdanau attention, trained for 250 steps (a little less than before).

Compare with the table in §3.5, where the model without attention reached a greedy token accuracy of about 0.5 at length 8 and almost never reversed a sequence exactly. With attention, and fewer training steps, length 8 is mostly solved and length 12 roughly doubles its greedy accuracy. The attention matrix for one length-8 input shows the mechanism. The weights form a band along the anti-diagonal. The first output step reads mostly the last source position, and the last output step reads mostly the first two, which is the alignment that reversal requires. The band is about two positions wide and leans one position to the right, because each bidirectional annotation also carries information about its neighbors, so reading a neighbor works almost as well. Nobody told the model about alignment. It learned it because looking in the right place lowered the loss.

Alignment matrices

Stack the weights $\alpha_{ij}$ for all target words $i$ and source words $j$ and you get a $T_y \times T_x$ alignment matrix. It is a soft version of the word alignments that classical statistical translation learned with separate models (the IBM alignment models of Brown et al. 1993), here learned jointly with translation and without any alignment labels. Bahdanau et al.'s alignment plots showed mostly monotonic alignments for English and French, with clean local reorderings where adjectives and nouns swap.

The widget shows an alignment matrix of the kind a trained model produces, with weights written by hand for illustration (the PyTorch block above learned a real one), for a sentence with two adjective-noun swaps and one English verb that becomes three French words. Click rows to see each context vector's recipe, change the softmax temperature to see soft versus hard alignment, and switch to the model without attention.

Without attention every row is identical, all weight on the final encoder state, because that is the only thing the decoder ever receives. As the temperature goes to zero each attention row becomes one-hot on its highest-scoring source word, a hard lookup, and as it grows each row flattens toward the uniform average, which throws the alignment away. The useful regime is in between, where attention is a differentiable, soft lookup. That view of attention as a soft dictionary lookup is exactly where §4.1 begins.

As $\tau$ shrinks, the weights concentrate on the matching annotation and the context vector converges to it. Attention interpolates between averaging everything and retrieving one item, and the network learns where on that spectrum each step should be.

Luong's multiplicative attention

A year later, Luong, Pham, and Manning (2015) simplified and systematized attention. They made three changes.

First, they score with the current decoder state. The decoder RNN first computes $\mathbf s_i$ from its input, then attends, then predicts, whereas Bahdanau attends with the previous state $\mathbf s_{i-1}$ and feeds the context into the RNN. Second, they tried several score functions, written here in our notation with $\mathbf h_j$ the encoder state,

The dot score needs no parameters but requires $\mathbf s_i$ and $\mathbf h_j$ to have the same size. The general score is a learned bilinear form, and it is the direct ancestor of the query-key product of Transformers (with $\mathbf W_a = \mathbf W_Q\mathbf W_K^\top$ when the projections are stored as $d \times d_k$ matrices, as §4.2 will show). The concat score is Bahdanau's additive score. Because the first two multiply the two vectors, they are called multiplicative attention.

Third, after computing the context $\mathbf c_i = \sum_j \alpha_{ij}\mathbf h_j$, they combine it with the decoder state into an attentional hidden state and predict from that,

They also fed $\tilde{\mathbf s}_i$ into the decoder's input at the next step (input feeding), so the decoder knows what it attended to last time, which helps it avoid translating the same word twice.

In plain words, Luong attention is the same three steps (score, normalize, read) with a cheaper score, placed after the RNN step instead of before it. The cost difference is real. For a sentence of $T_x$ words and $T_y$ decoder steps, the dot score costs $O(T_x T_y d)$, the general score the same once $\mathbf W_a\mathbf h_j$ is precomputed, and the additive score $O(T_x T_y n')$ plus a tanh for every pair. The dot product is also a single batched matrix multiply, which is what GPUs do best.

They also proposed local attention, which only looks at a window $[p_i - D, p_i + D]$ around a predicted source position $p_i = T_x\,\sigma(\mathbf v_p^\top\tanh(\mathbf W_p\mathbf s_i))$, multiplying the weights by a Gaussian $\exp\!\big(-(j - p_i)^2 / (2\sigma^2)\big)$ with $\sigma = D/2$ (they used $D = 10$). Local attention is cheaper for long inputs, an early example of the windowed attention patterns of §12.6.

Luong et al. found that the simple dot score worked well for global attention and the general score was better for local attention. An ensemble of their attention models set a new state of the art on WMT'15 English to German at 25.9 BLEU, and on WMT'14 English to German at 23.0, and local attention alone gave a gain of 5.0 BLEU over non-attentional systems.

import numpy as np

def luong_scores(s, H, kind, Wa=None, va=None):
    return np.einsum("bd,btd->bt", s, H)              # only handles "dot"

def attentional_state(c, s, Wc):
    return np.tanh(np.concatenate([s, c], -1) @ Wc)   # wrong order: the paper concatenates [c; s]

import numpy as np

def attn_decoder_step(s_prev, y_prev, H, P):
    raise NotImplementedError

From RNN plus attention to attention only

Look at what the recurrence is still doing in an attention model. The decoder's access to the source no longer depends on it, since attention provides that directly. What the RNNs still provide is context within each sentence, the biRNN making each annotation aware of its neighbors and the decoder RNN tracking what has been written so far. And they still impose their old costs. Each step must wait for the previous one, so a sentence of $n$ tokens takes $n$ sequential steps to encode, and information between distant words within the source still travels through $O(n)$ recurrent steps.

So the natural question was whether attention could provide the within-sentence context too. Several groups tried in 2016 and 2017. Cheng, Dong, and Lapata (2016) added intra-attention over a sentence's own earlier states inside an LSTM. Parikh et al. (2016) built a natural language inference model from attention between and within sentences with no recurrence at all, and it reached state-of-the-art accuracy with almost an order of magnitude fewer parameters than previous models. Lin et al. (2017) built sentence embeddings by letting a sentence attend over its own hidden states. The step that every word attends to every other word in its own sentence is called self-attention.

Then Vaswani et al. (2017) removed the recurrence entirely. In Attention Is All You Need, the encoder is a stack of self-attention layers over the source, the decoder is a stack of self-attention layers over the target written so far (with a mask, so that no position sees the future) plus attention over the encoder, exactly the encoder-decoder attention of this section. Their comparison of layer types is the cleanest summary of why it won, with $n$ the sequence length and $d$ the representation size.

Self-attention does more total work when sequences are long, but all of it can be done in parallel, and any two words are one step apart. On hardware built for large parallel matrix multiplies, that trade was overwhelming. It is also the trade that later chapters spend effort winning back, with the memory-efficient attention of FlashAttention and with recurrent models that can be trained in parallel, the linear-time alternatives that revisit this very table.

The last exercise is a first taste. Take a set of vectors and let each one attend to all of them, using the vectors themselves as queries, keys, and values. No recurrence, no parameters.

import numpy as np

def self_attend(X):
    S = X @ X.T
    A = np.exp(S) / np.exp(S).sum(0, keepdims=True)    # wrong axis, no scaling, unstable
    return A @ X

That second test is worth a pause. Shuffle the input words and the outputs shuffle the same way, unchanged. This attention layer has no idea what order its inputs came in, which for language is a serious problem, and it uses the same vector to ask, to be matched, and to answer. It also squares $n$ in its cost, and when a decoder uses it to generate text, each position must be prevented from attending to the words it has not written yet.

Where this leaves us

This chapter started from a broken promise, a word vector that could not read its sentence, and every section moved one step closer. Prediction gave us an objective that forces reading (§3.1). Embeddings let predictions generalize (§3.2). Recurrence carried context of any length, and gates kept its gradients alive (§3.3). Hidden states of language models became contextual word vectors (§3.4). Translation exposed the last weakness, everything squeezed through one vector (§3.5). Attention removed it, by letting every output read every input directly.

Strip away the translation setting and look at what is left. Attention is one operation in three steps. A query scores every stored vector, a softmax turns the scores into weights, and the output is the weighted average. You now hold everything this chapter can say about it, the additive and multiplicative scores, the temperature that slides it from a plain average to a hard lookup, padding masked with minus infinity before the softmax, and a self-attention with no parameters in which every vector reads all the others and order is invisible.

So here is the precise question attention answers. How can every position get the information it needs from any other position, in one step and in parallel, with the choice of what to read learned by gradient descent? The RNN's answer was to pass everything through one state, one step at a time, and this chapter measured what that costs. Attention's answer is a lookup by content that is soft enough to differentiate.

The version you wrote in Exercise 3.6d, $\operatorname{softmax}(\mathbf X\mathbf X^\top/\sqrt d)\,\mathbf X$, is not yet a layer that could replace an RNN. Each vector has to ask, be matched, and answer with the same numbers, and it has no parameters to learn what to ask for. The $\sqrt d$ is asserted, not derived. Nothing stops a word from reading the future, or a sentence from reading its padding. Each position gets one pattern of attention when it may need several, and the layer cannot tell "dog bites man" from "man bites dog". Chapter 4 starts exactly here. It first treats attention as a soft lookup in a table of keys and values and shows why the soft version can be learned when the hard one cannot (§4.1), then gives the three roles their own learned projections (§4.2), derives the scaling (§4.3), adds masks (§4.4) and several heads (§4.5), and studies the permutation symmetry you just met (§4.6) and what attention costs (§4.7). Word order comes back with the position encodings of Chapter 5.

Read more

• Bahdanau, Cho, and Bengio (2015). Neural Machine Translation by Jointly Learning to Align and Translate. The paper that introduced attention. Section 3 and Appendix A give the full model.
• Luong, Pham, and Manning (2015). Effective Approaches to Attention-based Neural Machine Translation. Global and local attention, the dot and general scores, and input feeding.
• Vaswani et al. (2017). Attention Is All You Need. Recurrence removed. Table 1 is the complexity comparison above, and the rest is Chapters 4 and 5.
• Parikh, Täckström, Das, and Uszkoreit (2016). A Decomposable Attention Model for Natural Language Inference, and Cheng, Dong, and Lapata (2016), Long Short-Term Memory-Networks for Machine Reading. Attention within a sentence, just before the Transformer.
• Jain and Wallace (2019). Attention is not Explanation, and Wiegreffe and Pinter (2019), Attention is not not Explanation. The debate on reading attention maps.
• Lilian Weng (2018). Attention? Attention! A clear survey of the attention family from Bahdanau to the Transformer.

Watch next

▶

▶

▶