Chinchilla: Estimating Training FLOPs
Beginner
The problem, from first principles
Hoffmann et al. (2022) ask how to split a fixed compute budget between model size and training data. The first ingredient is an estimate of how much compute a training run costs, which depends on the number of parameters and the number of tokens seen.
From theory to code
Implement training_flops(n_params, n_tokens), returning 6 * N * D.
Constraints
Nis parameters andDis training tokens.
Hints
Hint 1
Multiply the parameter count, the token count, and the constant 6.
Keywords
research-paperstransformersllmscalingcompute
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.