Batch Normalization: The Training-Time Forward Pass
Intermediate
The problem, from first principles
Activations drift in scale and offset as earlier layers change during training. Batch normalization (Ioffe & Szegedy, 2015) renormalizes each feature across the mini-batch, which stabilizes and speeds up training.
From theory to code
Implement batchnorm_train(x, gamma, beta, eps), which normalizes each feature with the batch mean and variance, then applies the learned scale and shift. Return the output along with the mean and variance it used.
Constraints
xhas shape(N, D).- Use the biased (population) variance, dividing by
N.
Hints
Hint 1
Compute mean and var over axis 0, normalize with sqrt(var + eps), then apply gamma * x_hat + beta.
Keywords
research-papersnormalizationbatchnorm
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.