05-gelu and 06-swish each build a self-gated activation from a different "how much to let through" curve, a Gaussian CDF for one, a sigmoid for the other. Mish is the third and last self-gated activation in this track, and it takes a different route to the same idea: instead of gating x directly with a probability curve, it first smooths x itself, then gates the smoothed version.
Concretely, Mish starts from softplus, a version of ReLU with the kink at 0 sanded off, and squashes that through tanh before multiplying back into x. The result is, once again, smooth everywhere, unbounded for large positive x, and slightly negative for a narrow range of negative inputs, the same family resemblance GELU and Swish share, built here entirely out of two functions this track already has in place: 03-tanh and the softplus function underlying 02-sigmoid.
Theory composes two already-familiar pieces, softplus (provided as _softplus) and tanh, then gates x with the result. Implement mish_forward(x) as that direct composition, and mish_backward(grad_output, x) from the chained product-rule derivative Theory works out, which reuses 03-tanh's own backward formula and _sigmoid as pieces of a larger expression.
x: any shape, elementwise operation._softplus (numerically stable np.logaddexp(0, x), not a literal log(1 + exp(x))) and _sigmoid helpers rather than re-deriving them.mish_backward(grad_output, x) receives the original input x, not mish_forward(x)'s output, same requirement as 05-gelu and 06-swish.x or grad_output in place.Open one at a time. Each gives away a little more than the last.
Compute _softplus(x) first, then np.tanh of that result, that intermediate value (call it sp, or its tanh, call it t) is the piece both forward and backward build on.
mish_backward needs 03-tanh's own derivative pattern, 1 - tanh(...)^2, applied to t = tanh(softplus(x)), as one factor, not as the whole answer.
The full local derivative is t + x * (1 - t**2) * sigmoid(x): the first term comes from differentiating the x factor (treating the tanh(softplus(x)) gate as constant), the second from differentiating the gate itself, chaining 03-tanh's derivative through softplus's own derivative, which happens to just be sigmoid(x).
Click "Run Tests" to test your implementation