GELU: The Derivative
Advanced
The problem, from first principles
Training needs the gradient of GELU, not just its value. Because GELU is a product of x and the Gaussian CDF, the derivative has a simple closed form that backpropagation uses directly.
From theory to code
Implement gelu_derivative(x), returning Phi(x) + x * phi(x) element-wise.
Constraints
phi(x) = exp(-x^2 / 2) / sqrt(2 pi).
Hints
Hint 1
Apply the product rule to x * Phi(x): the derivative is Phi(x) plus x times the density phi(x).
Keywords
research-papersactivationgelu
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.