Batch Normalization: Updating Running Statistics
Beginner
The problem, from first principles
Inference needs stable statistics, but a single batch is noisy. The paper (and later practice) tracks an exponential moving average of batch means and variances over training, updated after each batch.
From theory to code
Implement update_running_stats(running_mean, running_var, batch_mean, batch_var, momentum), which returns the updated running mean and variance.
Constraints
momentumis in[0, 1]; the default is0.1.
Hints
Hint 1
Blend the old value with the new one: (1 - momentum) * old + momentum * new, for both mean and variance.
Keywords
research-papersnormalizationbatchnorm
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.