RNN Regularization: Perplexity
Beginner
The problem, from first principles
Zaremba et al. (2014) regularize LSTM language models with dropout on the non-recurrent connections, and report results in perplexity. Perplexity is the exponential of the average per-token loss, so it reads as an effective number of choices per token.
From theory to code
Implement perplexity(total_nll, n), the exponential of the average negative log-likelihood per token.
Constraints
- Losses are in nats.
Hints
Hint 1
Divide the total loss by the token count, then exponentiate.
Keywords
research-paperssequence-modelslanguage-modellingregularization
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.