Stretch: bidirectional RNN
The problem, from first principles
[01-rnn-cell-forward]'s RNN processes a sequence strictly left to right: the hidden state at position 5 has seen positions 0 through 5, but has absolutely no information about position 6, 7, or beyond, they haven't happened yet, as far as the recurrence is concerned. For GENERATING text one token at a time (covered later in this curriculum), this causal, left-to-right constraint is exactly correct and unavoidable, you genuinely can't see the future when generating it. But for many other tasks, understanding a sentence's grammar, tagging each word's part of speech, translating a full sentence you already have complete access to, restricting the model to only ever see what came BEFORE each position is an artificial and unnecessary limitation: the word "bank" means something different in "I sat by the river bank" versus "I deposited money at the bank," and that disambiguating context can come from EITHER direction, sometimes from words that appear LATER in the sentence.
A bidirectional RNN removes this restriction directly: run TWO completely separate RNNs over the same sequence, one processing left-to-right as usual, one processing right-to-left, each with its own independently-learned weights, and at every position, concatenate BOTH directions' hidden states together. The result: every single position's final representation has now seen the ENTIRE sequence, both what came before it and what comes after it, something neither direction alone could ever provide.
From theory to code
Implement bidirectional_rnn_forward, running [01-rnn-cell-forward]'s rnn_cell_forward (already provided, reused via load_solution) TWICE across x_seq (shape (seq_len, batch_size, input_size)): once forward, stepping through positions 0 to seq_len-1 in order, and once backward, stepping through positions seq_len-1 down to 0, using its OWN separate weights. At every ORIGINAL time step, concatenate that step's forward-direction hidden state with that SAME step's backward-direction hidden state, and return the full (seq_len, batch_size, 2*hidden_size) result, in the ORIGINAL, un-reversed time order.
Constraints
- The forward pass must process
x_seqin order (t = 0, 1, ..., seq_len-1), usingweight_ih_fwd/weight_hh_fwd/etc. - The backward pass must process
x_seqin REVERSE order (t = seq_len-1, ..., 1, 0), using its OWN, separateweight_ih_bwd/weight_hh_bwd/etc. - The final result must be in ORIGINAL time order: position
t's output is the concatenation of the forward pass's hidden state AT positiontand the backward pass's hidden state AT positiont(not the order those values happened to be COMPUTED in during the backward loop). - The output's last axis must be
2*hidden_size: forward hidden state concatenated with backward hidden state, in that order.
Hints
Hint 1: The forward pass
h_fwd = h0_forward, then for t in range(seq_len): h_fwd = rnn_cell_forward(x_seq[t], h_fwd, weight_ih_fwd, weight_hh_fwd, bias_ih_fwd, bias_hh_fwd); forward_outputs.append(h_fwd), appending each step's output in the natural order they're computed.
Hint 2: The backward pass, and putting it back in order
h_bwd = h0_backward, then for t in reversed(range(seq_len)): h_bwd = rnn_cell_forward(x_seq[t], h_bwd, weight_ih_bwd, weight_hh_bwd, bias_ih_bwd, bias_hh_bwd); backward_outputs.append(h_bwd). This loop APPENDS in the order positions were VISITED (last position first), so call backward_outputs.reverse() afterward to put the list back into original time order before combining.
Hint 3: Combining
combined = [np.concatenate([f, b], axis=-1) for f, b in zip(forward_outputs, backward_outputs)], then return np.stack(combined, axis=0). Since both lists are now in original time order (after the reverse in Hint 2), zip correctly pairs up each position's forward and backward hidden states.
Click "Run Tests" to test your implementation