Seq2seq: The Sequence Negative Log-Likelihood
Intermediate
The problem, from first principles
The seq2seq model is trained to maximize the probability of the correct target sentence. Because a sentence probability is a product of per-word probabilities, its negative log is a sum of per-word losses.
From theory to code
Implement sequence_nll(probs, targets), which returns the summed negative log-probability of the correct token at each step.
Constraints
probshas shape(T, V), one distribution per step.
Hints
Hint 1
Pick the probability of each correct token with fancy indexing, take -log, then sum.
Keywords
research-paperstransformersllmseq2seq
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.