Dropout fwd/bwd

The problem, from first principles

A large network with millions of parameters can, during training, start relying heavily on a small number of specific neurons acting in a very particular coordinated way, a kind of overfitting where different neurons become "co-adapted" to compensate for each other's specific quirks rather than each independently learning something generally useful. Dropout is a strikingly simple fix: on every training step, randomly zero out a fraction of the neurons' activations entirely, forcing every OTHER neuron to be independently useful, since it can no longer count on any particular teammate being present on the next step. At test time, all neurons participate (no zeroing), so the network gets to use its FULL capacity when it actually matters.

There's a subtlety in exactly how the surviving activations get treated, and it's the crux of this question: if training randomly zeroes out a fraction p of activations but test time uses ALL of them, the total activation magnitude flowing into the next layer would systematically differ between training and testing, which would itself hurt the network's performance. "Inverted dropout" fixes this by rescaling the SURVIVING activations during training (by 1 / (1 - p)) so their expected magnitude matches what test time will see, meaning test time needs NO special-casing at all.

From theory to code

Implement dropout_forward(x, mask, p) and dropout_backward(grad_output, mask, p). mask is a same-shape 0/1 array already generated by the caller (1 = keep this entry, 0 = zero it out); you are not responsible for generating the random mask itself, only for correctly applying it with the inverted-dropout rescaling.

Constraints

• Every entry where mask == 0 must be exactly 0 in the output.
• Every entry where mask == 1 must be scaled by 1 / (1 - p), not left unscaled.
• dropout_backward must apply the identical mask and identical scaling as dropout_forward did (dropout has no learnable parameters, but it still needs a backward pass so gradients continue flowing correctly to whatever comes before it in the network).
• Assume 0 <= p < 1.

Hints