Attention Is All You Need: Sinusoidal Positions
Intermediate
The problem, from first principles
A transformer has no recurrence, so it needs another way to know word order. The paper adds fixed sine and cosine waves of different frequencies to the embeddings. Each position gets a unique pattern, and the model can learn to attend by relative offset.
From theory to code
Implement sinusoidal_positional_encoding(T, d), which returns the (T, d) table of sines and cosines.
Constraints
dis even.
Hints
Hint 1
Build the angle matrix pos / 10000^(2i/d), then fill even columns with sin and odd columns with cos.
Keywords
research-paperstransformersllmattentiontransformer
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.