Stacking linear layers on top of each other, with nothing in between, is still just one linear layer — matrix multiplications compose into another matrix multiplication, no matter how many you chain. A network needs a nonlinearity between its linear layers to represent anything more expressive than a straight line, and ReLU is the simplest one that still works well in practice: it passes positive signals through untouched and kills negative ones outright.
Implement relu_forward(x), the elementwise max(0, x), and relu_backward(grad_output, x), which needs to know which elements of the original input x were positive in order to decide how much of the incoming gradient to let through at each position.
x, grad_output: any matching NumPy array shape.relu_forward returns the same shape as x, every entry either 0 or the original positive value.relu_backward takes the incoming grad_output and the original forward-pass input x (not the forward output), and returns a same-shape gradient.relu_backward's gradient at exactly x == 0 is 0 (PyTorch's own convention, one of several mathematically defensible choices at a non-differentiable point, but the one real PyTorch uses).x or grad_output.Open one at a time. Each gives away a little more than the last.
relu_forward is one NumPy call: an elementwise maximum against 0.
relu_backward needs a 0/1 mask of "was this input positive," the same shape as x — build it with a comparison, then multiply it into grad_output.
Click "Run Tests" to test your implementation