RoFormer: The Rotary Frequencies
Intermediate
The problem, from first principles
RoFormer (Su et al., 2021) encodes position by rotating query and key vectors, rather than adding a position vector. Each pair of features rotates at its own frequency, so the dot product between a rotated query and key depends on how far apart their positions are.
From theory to code
Implement rope_frequencies(d, base), returning the d / 2 rotation frequencies.
Constraints
dis even.
Hints
Hint 1
Compute base ** (-2 i / d) for i in 0 .. d/2 - 1 as one vector.
Keywords
research-paperstransformersllmpositionrope
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.