Layer Normalization: The Learned Scale and Shift
Beginner
The problem, from first principles
Plain normalization removes information the layer may need, such as an overall scale or offset per feature. Layer norm therefore adds a learned gain and bias, as in the paper, so the layer can choose how much normalization to keep.
From theory to code
Implement layernorm(x, gamma, beta, eps), which normalizes each example over its features and then applies the per-feature gamma and beta.
Constraints
gammaandbetahave shape(D,).
Hints
Hint 1
Reuse the last-axis normalization from normalize_last_axis and multiply by gamma, then add beta.
Keywords
research-papersnormalizationlayernorm
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.