Scaling Laws: The Power-Law Loss
Beginner
The problem, from first principles
Kaplan et al. (2020) found that language model loss falls as a power law in model size, dataset size and compute, across many orders of magnitude. A power law is a straight line on a log-log plot, which is why it is easy to fit and extrapolate.
From theory to code
Implement power_law_loss(N, N_c, alpha), which returns (N_c / N) ** alpha.
Constraints
- Works on scalars and arrays.
Hints
Hint 1
Divide the scale by the size and raise to the exponent.
Keywords
research-paperstransformersllmscaling
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.