GPT-3: The Causal Attention Mask
Beginner
The problem, from first principles
A GPT-style model is trained to predict each token from the tokens before it. During training every position is processed at once, so attention must be blocked from looking at the future. A causal mask does that.
From theory to code
Implement causal_mask(n), returning an n x n boolean matrix that allows attention only to current and earlier positions.
Constraints
Truemeans the attention is allowed.
Hints
Hint 1
Use a lower-triangular matrix of ones, cast to boolean.
Keywords
research-paperstransformersllmllmtransformer
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.