Attention Is All You Need: Splitting Into Heads
Beginner
The problem, from first principles
Multi-head attention runs several attention functions in parallel, each on a smaller slice of the features. Splitting the vectors into heads is the first step, and it costs no extra parameters.
From theory to code
Implement split_heads(x, h), which turns a (T, d) sequence into (h, T, d // h).
Constraints
dmust be divisible byh.
Hints
Hint 1
Reshape to (T, h, d // h), then transpose so the head axis comes first.
Keywords
research-paperstransformersllmattentiontransformer
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.