Chinchilla: Choosing Model Size for a Budget
Intermediate
The problem, from first principles
If you have a fixed compute budget, how big should the model be? Combining the two rules gives a square-root relationship: the compute-optimal parameter count grows with the square root of the budget.
From theory to code
Implement optimal_params_for_budget(flops), returning the compute-optimal parameter count N = sqrt(C / 120).
Constraints
- Derive from
C = 6 N DwithD = 20 N.
Hints
Hint 1
Substitute D = 20N into C = 6ND to get C = 120 N^2, then solve for N.
Keywords
research-paperstransformersllmscalingcompute
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.