He Initialization: The Standard Deviation Rule
Beginner
The problem, from first principles
Deep ReLU networks lose signal scale layer by layer. ReLU zeroes half of its inputs, so the variance halves at each layer unless the weights are scaled to compensate.
From theory to code
Implement he_init_std(fan_in), the standard deviation of the weight distribution for a layer with fan_in inputs.
Constraints
fan_inis a positive integer.
Hints
Hint 1
Variance should be 2 / fan_in, so the standard deviation is its square root.
Keywords
research-papersinitializationrelu
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.