RoFormer: The Dot Product Depends Only on Relative Position

The problem, from first principles

The point of RoPE is that attention scores should depend on how far apart two tokens are, not on their absolute positions. Rotating queries and keys by their own positions gives exactly that: the product of two rotations depends only on the difference of the angles.

From theory to code

Implement rope_dot(q, k, m, n, base), which rotates the query to position m, the key to position n, and returns their dot product.

Constraints

• Equal positions reduce to the ordinary dot product.

Hints