He Initialization: Sampling a Weight Matrix
Intermediate
The problem, from first principles
A weight matrix sampled naively, with the same scale for every layer, makes activations explode or vanish with depth. The initialization must match the layer's width and activation function.
From theory to code
Implement he_init_weights(fan_in, fan_out, rng), which returns a (fan_out, fan_in) matrix sampled with He's standard deviation.
Constraints
rngis anumpy.random.Generator.- The result has shape
(fan_out, fan_in).
Hints
Hint 1
Use rng.normal(0, std, size=...) with std = sqrt(2 / fan_in).
Keywords
research-papersinitializationrelu
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.