Assemble one full block (attention, norm, residual, FFN, norm, residual)

The problem, from first principles

Every piece needed to build a real Transformer block already exists somewhere in this curriculum: [04-seq-modeling/04-attention/05-mha-concat-output-projection]'s Multi-Head Attention lets every position gather information from every other position; [01-layer-normalization-forward]'s LayerNorm keeps activations at a stable scale as they flow through many stacked layers; [03-residual-connection]'s skip connections keep gradients flowing cleanly through depth; [04-feedforward-sublayer]'s FFN gives each position room for genuine per-position nonlinear computation. A Transformer block is nothing more than these four pieces, wired together in a SPECIFIC order, applied twice (once around attention, once around the FFN).

The specific ordering matters. This question implements the "Pre-Norm" arrangement (used by GPT-2 and most modern LLMs): LayerNorm is applied to the block's input BEFORE each sublayer runs, and the residual connection always adds the sublayer's output back onto the RAW (un-normalized) input, never the normalized version. [05-transformers-llm/02-modern-transformer-architecture]'s "Pre-norm vs. post-norm" question, immediately following this track, examines the alternative ("Post-Norm," the original 2017 Transformer paper's choice) and WHY the ordering has a real, measurable effect on how easy the resulting network is to train.

From theory to code

Implement transformer_block_forward, composing the block from already-built functions:

normed1  = LayerNorm(x, gamma1, beta1)
attn_out = MultiHeadAttention(normed1, normed1, normed1, num_heads, weight_o, bias_o, mask)
x1       = x + attn_out                                    # residual onto the RAW x, not normed1

normed2  = LayerNorm(x1, gamma2, beta2)
ffn_out  = FeedForward(normed2, ffn_weight1, ffn_bias1, ffn_weight2, ffn_bias2)
x2       = x1 + ffn_out                                     # residual onto x1, not normed2

return x2

Self-attention means the SAME normalized tensor is passed as query, key, AND value.

Constraints

• Two INDEPENDENT LayerNorms, each with its own gamma/beta: one before attention, one before the FFN. They are never shared.
• Each residual connection adds a sublayer's output back onto the tensor that FED INTO that sublayer's LayerNorm (its own "raw" input), never onto the normalized version.
• Self-attention: query = key = value = the first LayerNorm's output.
• mask (when given) passes straight through to MultiHeadAttention, unchanged.

Hints