Encoder vs. decoder vs. encoder-decoder: three ways to arrange the same block

The problem, from first principles

[01-transformer-block/06-assemble-full-block] built ONE reusable Transformer block. Nothing about that block's internal STRUCTURE changes across "BERT," "GPT," and "the original 2017 Transformer" (translation-style encoder-decoder models), the three big architectural families real systems fall into. What changes is entirely WHICH POSITIONS self-attention is allowed to see, and, for the encoder-decoder family, whether a SECOND kind of attention gets added at all.

An encoder (BERT-style) lets every position attend to every other position, in both directions: useful for building a representation of a COMPLETE, already-fully-visible input (classification, understanding tasks), where there's no reason to hide any part of the input from any other part. A decoder (GPT-style) restricts self-attention with [04-seq-modeling/04-attention/02-causal-mask]'s causal mask: essential for autoregressive generation, where a model must never "cheat" by looking at tokens it hasn't generated yet. An encoder-decoder (translation-style) uses BOTH: an encoder processes the full source sequence bidirectionally, and a decoder generates the target sequence causally, while ALSO attending to the encoder's output via a second, distinct attention call, "cross-attention," where the query comes from the decoder but the key/value come from the encoder.

From theory to code

Implement encoder_block_forward (a thin wrapper around [06-assemble-full-block]'s block, no mask), decoder_block_forward (the same block, with [02-causal-mask]'s mask), and encoder_decoder_cross_attention (a direct call to [05-mha-concat-output-projection]'s multi_head_attention, with query from the decoder and key/value both from the encoder's output).

Constraints

• encoder_block_forward passes mask=None to the underlying Transformer block: full bidirectional attention.
• decoder_block_forward builds a causal mask sized to x's own sequence length and passes it through.
• encoder_decoder_cross_attention's query is decoder_hidden; its key AND value are both encoder_output. These may have DIFFERENT sequence lengths (the source and target sentences in translation are rarely the same length).
• Cross-attention uses NO mask (the decoder is always allowed to attend to the entire, already-fully-computed encoder output).

Hints