LLaMA: Choosing the Feed-Forward Width
Intermediate
The problem, from first principles
SwiGLU has three weight matrices instead of two, so to keep the parameter count matched to a standard feed-forward layer, the hidden width is set to about two thirds of four times the model width. The result is rounded up to a hardware-friendly multiple.
From theory to code
Implement llama_hidden_dim(dim, multiple_of), returning int(8 * dim / 3) rounded up to a multiple of multiple_of.
Constraints
- The default multiple is 256.
Hints
Hint 1
Compute int(2 * 4 * dim / 3), then round up with integer arithmetic.
Keywords
research-paperstransformersllmfeedforwardllama
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.