LLaMA: The SwiGLU Feed-Forward Gate
Intermediate
The problem, from first principles
The feed-forward layer in LLaMA multiplies two projections of the input, one of them passed through a SiLU (Swish) nonlinearity. This gated linear unit, called SwiGLU, performs better than the plain ReLU MLP at similar size.
From theory to code
Implement swiglu(x, W, V), returning silu(x W) * (x V).
Constraints
silu(z) = z * sigmoid(z).
Hints
Hint 1
Project with W, apply silu, project with V, and multiply element-wise.
Keywords
research-paperstransformersllmfeedforwardllama
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.