Vanilla RNN cell, forward
The problem, from first principles
[03-dl-training/02-layers/01-linear-forward]'s linear_forward processes a single, fixed-size input and produces a single output, it has no notion of "sequence" or "what came before" at all. Recurrent Neural Networks were the first widely-used architecture built specifically to process SEQUENCES: instead of a single forward pass, an RNN walks through a sequence one element at a time, and at each step, combines the CURRENT input with a running "memory" of everything seen so far (the "hidden state"), producing both an output for that step and an UPDATED memory to carry into the next step. This is precisely the "processes tokens one at a time IN ORDER" behavior [02-embeddings/03-sinusoidal-positional-encoding]'s Statement contrasted attention against: an RNN's very act of stepping through a sequence in order IS how it encodes position, no separate positional encoding scheme needed at all.
This question implements exactly ONE step of that process: given the current input and the previous hidden state, compute the new hidden state. Vanilla RNN cell, backward and Backprop through time (BPTT), immediately following this question, build on this single step to understand training an RNN across a whole sequence, and how gradients behave when this same step gets repeated many times in a row.
From theory to code
Implement rnn_cell_forward(x, h_prev, weight_ih, weight_hh, bias_ih, bias_hh). Compute a linear transformation of the CURRENT input (x @ weight_ih.T + bias_ih) and a SEPARATE linear transformation of the PREVIOUS hidden state (h_prev @ weight_hh.T + bias_hh), sum both results together, and apply tanh to produce the new hidden state.
Constraints
weight_ihhas shape(hidden_size, input_size), matching[03-dl-training/02-layers/01-linear-forward]'s(out_features, in_features)convention;weight_hhhas shape(hidden_size, hidden_size)(since the hidden state feeds into itself).- Both linear transformations (input-to-hidden and hidden-to-hidden) must be computed and summed BEFORE applying
tanh, nottanhapplied to each separately and then summed. - The output has the SAME shape as
h_prev:(batch_size, hidden_size). - Must work correctly for any
batch_size, includingbatch_size == 1.
Hints
Hint 1
input_contribution = x @ weight_ih.T + bias_ih, exactly [03-dl-training/02-layers/01-linear-forward]'s linear_forward applied to x.
Hint 2
hidden_contribution = h_prev @ weight_hh.T + bias_hh, the SAME kind of linear transformation, applied to h_prev instead of x, with its own separate weight/bias.
Hint 3
return np.tanh(input_contribution + hidden_contribution): sum both contributions FIRST, then apply tanh once to the combined sum, not tanh to each piece separately.
Click "Run Tests" to test your implementation