Stretch: SwiGLU-gated FFN

The problem, from first principles

[04-feedforward-sublayer]'s "expand, activate, project" FFN applies the SAME activated projection to every hidden unit, with no way for the network to modulate, per-input, HOW MUCH of each hidden unit's signal should actually pass through. Shazeer (2020) proposed a GATED variant, borrowing the "gate" idea from LSTMs/GRUs ([04-seq-modeling/03-recurrent-neural-networks]): compute TWO separate linear projections of the input instead of one, pass ONE of them through an activation to act as a per-unit GATE (a soft "how much of this feature matters right now" signal, similar in spirit to an LSTM's forget/input gates), and multiply it elementwise against the OTHER (left un-activated). This gives the sublayer a genuinely richer function class than a plain activated MLP, letting different hidden units get dynamically, INPUT-DEPENDENTLY suppressed or amplified, and LLaMA, PaLM, Mistral and most other modern LLMs adopted this variant specifically because it measurably improves model quality for roughly the same parameter and compute budget as the plain FFN.

"SwiGLU" names the specific choice of gating activation: Swish ([02-deep-learning-core/02-activations/06-swish], also called SiLU) applied to the gate branch, GLU (Gated Linear Unit) naming the elementwise-multiplication-of-two-projections pattern itself.

From theory to code

Implement swiglu_ffn(x, weight_gate, weight_up, weight_down). Compute TWO separate linear projections of x up to d_ff, using weight_gate and weight_up respectively (both WITHOUT a bias term, matching real LLaMA-style implementations, which drop biases from every linear layer). Pass the weight_gate projection through [02-deep-learning-core/02-activations/06-swish]'s swish_forward, multiply it ELEMENTWISE by the (un-activated) weight_up projection, then project the result back down to d_model with weight_down (also bias-free).

Constraints

• weight_gate and weight_up are TWO SEPARATE weight matrices, each shape (d_ff, d_model), computing two genuinely different projections of the same input x.
• Only the weight_gate branch passes through swish_forward; the weight_up branch stays purely linear.
• The two branches combine via ELEMENTWISE MULTIPLICATION (gate * up), not addition, not concatenation.
• No bias terms anywhere: weight_gate, weight_up, and weight_down are all used without an additive bias, matching real modern LLM implementations.

Hints