Weight Normalization: Reparameterizing the Weights
Intermediate
The problem, from first principles
Weight normalization (Salimans & Kingma, 2016) splits each weight vector into a length g and a direction v / ||v||. Training then adjusts length and direction separately, which speeds up optimization.
From theory to code
Implement weight_norm(v, g), which builds the weight matrix with each row of length g[i] pointing along v[i].
Constraints
vhas shape(out, in),ghas shape(out,).
Hints
Hint 1
Divide each row of v by its norm, then multiply each row by its g value.
Keywords
research-papersnormalizationweight-norm
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.