Swish (SiLU) fwd/bwd

The problem, from first principles

05-gelu replaced ReLU's hard on/off gate with a probabilistic one built from the Gaussian CDF. That's one way to build a "self-gated" activation, where the input decides its own fate, but it's not the only one. Swish reaches the same kind of smooth, self-gating behavior using a function this track has already implemented from scratch: 02-sigmoid.

Instead of asking "how likely is x to be worth keeping" via a Gaussian, Swish asks the same question via a sigmoid: sigmoid(x) is already a number between 0 and 1 that grows with x, so using it directly as the "how much to let through" gate is a natural, cheaper alternative to GELU's Gaussian CDF. This is why Swish (Google Brain's name for it) and SiLU (PyTorch's name) are the exact same function under two names, they're built from the exact building block Swish's name describes: a sigmoid linear unit.

From theory to code

Theory writes Swish as x scaled by sigmoid(x), using the provided _sigmoid helper (the same clipped, overflow-safe formula from 02-sigmoid). Implement swish_forward(x) directly from that, then swish_backward(grad_output, x) from the product-rule derivative Theory works out, reusing 02-sigmoid's own derivative as one term.

Constraints

• x: any shape, elementwise operation.
• Use the provided _sigmoid helper rather than re-deriving a raw 1 / (1 + exp(-x)).
• swish_backward(grad_output, x) receives the original input x, not swish_forward(x)'s output, same requirement as 05-gelu.
• One vectorized expression per function, no explicit loop over elements.
• Neither function modifies x or grad_output in place.

Hints

Open one at a time. Each gives away a little more than the last.