RNN Regularization: Dropout on Non-recurrent Connections

The problem, from first principles

Zaremba et al. found that dropout works for LSTMs only when it is applied to the non-recurrent connections, between layers and before the output. Applying it to the recurrent state erases memory too fast.

From theory to code

Implement nonrecurrent_dropout(h, mask, p), the inverted dropout applied to a non-recurrent activation.

Constraints

• The mask is supplied by the caller.

Hints