GELU: The Tanh Approximation
Intermediate
The problem, from first principles
The exact GELU needs an error function, which is slow on some hardware. The paper also gives a tanh approximation that uses only multiplications, additions and tanh, and matches the exact curve closely.
From theory to code
Implement gelu_tanh(x) using 0.5 x (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))).
Constraints
- Must agree with the exact GELU to within
1e-3on[-3, 3].
Hints
Hint 1
Evaluate the polynomial inside tanh first, then multiply by 0.5 * x.
Keywords
research-papersactivationgelu
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.