Swish: The Derivative With Beta
Intermediate
The problem, from first principles
Training needs the gradient of Swish. Its derivative combines the gate and the input's own contribution, so it stays nonzero for negative inputs, unlike ReLU's zero gradient there.
From theory to code
Implement swish_derivative(x, beta), returning s + beta * x * s * (1 - s) with s = sigmoid(beta * x).
Constraints
- Must match a numerical derivative of
swish.
Hints
Hint 1
Write s = sigmoid(beta * x), then apply the product rule to x * s.
Keywords
research-papersactivationswish
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.