Residual/skip connection

The problem, from first principles

[03-dl-training]'s vanishing-gradient investigations, and [04-seq-modeling/03-recurrent-neural-networks/03-bptt-vanishing-exploding] most vividly, showed exactly why deep networks are hard to train: a gradient flowing backward through many stacked layers gets multiplied, over and over, by each layer's local Jacobian. Multiply by something a bit less than 1 fifty times in a row and the gradient vanishes to essentially nothing; multiply by something a bit more than 1 fifty times and it explodes. Either way, the earliest layers in a deep stack end up learning far more slowly, or far more unstably, than the later ones, purely as a side effect of DEPTH itself.

He et al. (2015, ResNet) introduced a strikingly simple fix: alongside each sublayer's transformation, keep an unmodified copy of the input flowing forward too, and ADD the two together, output = x + sublayer(x), rather than replacing x outright with sublayer(x). This "residual" or "skip" connection gives the gradient an alternate path back through the network that involves NO multiplication at all, only addition, and addition's local gradient is always exactly 1, regardless of depth. Every modern deep architecture, Transformers very much included, is built almost entirely out of blocks wrapped in residual connections, precisely because this is what makes stacking dozens or hundreds of layers actually trainable in practice.

From theory to code

Implement residual_connection(x, sublayer_output), a direct elementwise addition, and residual_connection_backward(grad_output), the backward pass through that addition, which the "Assemble one full block" question, later in this track, will wire directly into a Transformer block's attention and feed-forward sublayers.

Constraints

• residual_connection is a plain elementwise sum: x + sublayer_output, nothing more.
• residual_connection_backward returns (grad_x, grad_sublayer_output): BOTH gradients equal grad_output itself, unchanged (the local gradient of addition with respect to either input is always 1).
• Shapes of x and sublayer_output always match (the sublayer is designed to preserve x's shape exactly, so no broadcasting is needed here).

Hints