GELU: The Exact Gaussian-Gated Activation
Intermediate
The problem, from first principles
ReLU gates inputs with a hard threshold at zero. GELU (Hendrycks & Gimpel, 2016) gates each input by how likely it is to be positive under a Gaussian, giving a smooth version of that decision.
From theory to code
Implement gelu_exact(x), which returns x * Phi(x), where Phi is the standard normal cumulative distribution.
Constraints
- Uses
math.erf; no SciPy.
Hints
Hint 1
Phi(x) = 0.5 * (1 + erf(x / sqrt(2))). Multiply the input by that value.
Keywords
research-papersactivationgelu
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.