LARS: Per-layer Norms
Beginner
The problem, from first principles
LARS needs a norm per layer, not one global norm. A weight tensor's norm is taken over all its entries, which is the Frobenius norm for a matrix.
From theory to code
Implement layer_norms(layers), returning each layer's weight norm.
Constraints
- Layers are kept separate.
Hints
Hint 1
Map the Euclidean norm over the list of layer arrays.
Keywords
research-papersoptimizationlarge-batch
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.