Swish: A Self-Gated Activation
Beginner
The problem, from first principles
ReLU is cheap but has a hard kink at zero. The Swish paper (Ramachandran, Zoph & Le, 2017) found by automated search that x * sigmoid(x) often beats ReLU on deep networks, because the input gates itself smoothly.
From theory to code
Implement swish(x, beta), which returns x * sigmoid(beta * x).
Constraints
- With
beta = 1this is the standard Swish (also called SiLU).
Hints
Hint 1
The sigmoid is 1 / (1 + exp(-beta * x)); multiply it by x.
Keywords
research-papersactivationswish
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.