Identity Mappings in ResNet: The Gradient Through the Shortcut
Advanced
The problem, from first principles
The paper's central argument is about the backward pass. With an identity shortcut, the gradient reaching earlier layers always includes a copy of the upstream gradient, so it cannot vanish through the shortcut alone.
From theory to code
Implement identity_path_gradient(grad_out, jac_f), returning grad_out (I + J_F) for a row-vector gradient.
Constraints
jac_fis the(D, D)Jacobian of the residual branch.
Hints
Hint 1
Write grad_out @ (I + jac_f), which equals grad_out + grad_out @ jac_f.
Keywords
research-papersarchitectureresnet
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.