01-relu's gate is binary: a neuron's input either survives untouched (x > 0) or gets zeroed out completely (x <= 0), decided purely by sign. That's a crude rule. A value of x = 0.01 and a value of x = -0.01 sit almost on top of each other, yet ReLU treats one as "fully kept" and the other as "fully discarded."
GELU replaces that hard switch with a soft, probabilistic one: instead of asking "is x positive?", it asks "how likely is x to be a value worth keeping?" and scales x down by that likelihood. Values far from zero (very positive or very negative) get a decisive answer, close to "keep everything" or "keep nothing." Values near zero get partial credit. This smoother gating is one reason GELU, not ReLU, became the default activation inside BERT, GPT, and most transformer architectures built after them.
Theory expresses "how likely is x to be worth keeping" as the standard normal CDF, Phi(x), and defines GELU as x scaled by Phi(x). Phi itself is written in terms of the error function erf, which the file already imports and vectorizes as _erf.
Implement gelu_forward(x) using _erf to build Phi(x), then gelu_backward(grad_output, x) using the product-rule derivative Theory derives. Note the backward signature: it takes the original x, not a saved output, the "why" is covered in Theory.
x: any shape, elementwise operation._erf (a vectorized math.erf) rather than any other error-function implementation.gelu_backward(grad_output, x) receives the original input x, not gelu_forward(x)'s output, unlike 02-sigmoid, 03-tanh, and 04-softmax.x or grad_output in place.Open one at a time. Each gives away a little more than the last.
Phi(x), the standard normal CDF, shows up in both gelu_forward and gelu_backward. Write it once as its own expression using _erf, _SQRT_2, and it becomes a shared building block for both functions.
The backward pass needs the standard normal density phi(x) (lowercase), not just Phi(x) (uppercase). They're different functions: phi(x) = exp(-x^2/2) / sqrt(2*pi), _SQRT_2PI is already provided for exactly this.
gelu_backward combines Phi(x) and x * phi(x) by addition, then multiplies the sum by grad_output. That's the whole function, no other terms are needed.
Click "Run Tests" to test your implementation