Sequence-to-sequence / encoder-decoder: the bottleneck problem attention was invented to solve

The problem, from first principles

Machine translation (and many other tasks: summarization, question answering) needs to map one variable-length sequence to another. The classic pre-Transformer solution, "sequence-to-sequence" (Sutskever et al., 2014), is elegantly simple: an ENCODER RNN ([01-rnn-cell-forward], applied repeatedly, exactly [06-bidirectional-rnn]'s forward-direction loop) reads the entire input sentence and compresses it down into a single, FIXED-SIZE hidden state, the "context vector." A DECODER RNN then starts from that context vector and generates the output sentence, one token at a time.

This design has a genuine, well-documented weakness this question demonstrates numerically rather than just describing: hidden_size stays FIXED regardless of how long the input sentence is. A 5-word sentence and a 50-word sentence both get compressed down into the exact SAME size vector, and, just like [03-bptt-vanishing-exploding] demonstrated for gradients flowing backward, an RNN's hidden state, updated step by step going FORWARD, tends to be dominated by RECENT inputs, information from early in a long sequence can get substantially overwritten by everything that came after it, long before the encoder ever reaches the end. The FINAL hidden state, the only thing the decoder ever sees, disproportionately reflects the LATE part of the input, with early content potentially barely surviving at all. This is the exact motivation named directly in Scaled dot-product attention, the very next track in this curriculum: instead of forcing the ENTIRE input sequence through one fixed-size bottleneck, attention gives the decoder access to EVERY encoder hidden state directly, letting it look back at whichever parts of the input actually matter for the CURRENT word being generated, regardless of how far back in the sequence they occurred.

From theory to code

Implement encode_all_hidden_states(x_seq, h0, weight_ih, weight_hh, bias_ih, bias_hh), running [01-rnn-cell-forward]'s rnn_cell_forward (already provided, reused via load_solution) across the whole sequence and returning EVERY step's hidden state. Implement get_bottleneck_context(hidden_states), extracting just the LAST one, the classic seq2seq "context vector," everything else discarded. Implement cosine_similarity(a, b), a similarity measure between 0 (unrelated) and 1 (identical direction), used to numerically MEASURE how much information from different parts of a sequence actually survives into the final context vector.

Constraints

• encode_all_hidden_states must return ALL seq_len hidden states, shape (seq_len, batch_size, hidden_size), not just the last one.
• get_bottleneck_context returns exactly hidden_states[-1], the final time step's hidden state, discarding everything from earlier steps.
• cosine_similarity must correctly handle inputs of ANY matching shape (flatten both before computing), returning a single scalar float.

Hints