Attention Is All You Need: Scaled Dot-Product Attention
Intermediate
The problem, from first principles
Attention Is All You Need (Vaswani et al., 2017) drops recurrence entirely. Each position builds its output by comparing a query with every key, turning the similarities into weights, and averaging the values with those weights. Dividing by the square root of the key dimension keeps the softmax from saturating.
From theory to code
Implement scaled_dot_product_attention(Q, K, V), which returns softmax(Q K^T / sqrt(d)) V with a row-wise softmax.
Constraints
QandKshare the feature dimensiond.
Hints
Hint 1
Compute Q @ K.T / sqrt(d), apply a row-wise softmax with the max-subtraction trick, then multiply by V.
Keywords
research-paperstransformersllmattentiontransformer
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.