Notes
Your notebooks, with code that runs, boxes from the book, and papers with your notes.
Attention, in my words
Why the dk
Raw scores grow with the width of the keys. Scaled, they keep a spread of one, so no head starts out one-hot.
PDF
15 pages · 1 note
We suspect that for large values of dk, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.
The same argument as 4.3, in one line.
Page 4
▶ Run[1]4.3 Scaled dot-product attention
22.6 1.0
The largest weight per row, against the width of the keys
Interview question
What exactly goes wrong if you leave the scaling out?From 4.3 Scaled dot-product attentionScores have standard deviation √d_k, so with d_k = 128 the gaps between scores are around 10 and the softmax is nearly one-hot. Its Jacobian is then nearly zero, so almost no gradient reaches the query and key weights.Which widths make the unscaled softmax one-hot?
From your table, by d=256 the largest unscaled weight is already 1.00, so every row is one-hot. Scaled, it stays near 0.2 at every width.
Ask the assistant about this notebook…
Claude Sonnet 5.5
∇
Notes are part of Pro
Notebooks for code, passages from the book, and more.