LLaMA: RMS Normalization
Beginner
The problem, from first principles
LLaMA (Touvron et al., 2023) replaces LayerNorm with RMSNorm, a simpler normalization that rescales by the root mean square without subtracting the mean. It is cheaper and performs about as well.
From theory to code
Implement rmsnorm(x, g, eps), dividing each row by its root mean square and multiplying by the gain g.
Constraints
- No mean subtraction and no bias term.
Hints
Hint 1
Compute the mean of squares over the last axis, take the square root with eps, divide, then multiply by g.
Keywords
research-paperstransformersllmnormalizationllama
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.