RoFormer: Rotating a Vector by Position
Advanced
The problem, from first principles
A pair of features can be rotated in a plane. RoPE rotates each pair by an angle equal to the token's position times a fixed frequency, so every position is encoded directly in the geometry of the vectors.
From theory to code
Implement rope_rotate(x, pos, base), which rotates each consecutive pair of features by pos * theta_i.
Constraints
- Pairs are
(x[0], x[1]),(x[2], x[3]), and so on.
Hints
Hint 1
Compute the angles from the frequencies, then apply a 2D rotation to each pair of even and odd indices.
Keywords
research-paperstransformersllmpositionrope
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.