Layer Normalization: Per-Example Statistics
Intermediate
The problem, from first principles
The key difference from batch norm is which axis the statistics come from. Layer norm computes one mean and variance per example, across its features, so the result never depends on batch size.
From theory to code
Implement per_example_stats(x), which returns the mean and variance of each example's features as two arrays of length N.
Constraints
xhas shape(N, D).- Variance is biased (divides by
D).
Hints
Hint 1
Reduce over the last axis: mean(axis=-1) and var(axis=-1).
Keywords
research-papersnormalizationlayernorm
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.