BERT: The Masked Language Model Loss
Intermediate
The problem, from first principles
BERT's training loss is only computed on the masked positions. Unmasked tokens contribute nothing, so the model learns to recover the hidden words from their context on both sides.
From theory to code
Implement masked_lm_loss(log_probs, targets, mask), which averages the negative log-probability of the correct token over masked positions.
Constraints
- Return
0.0when no position is masked.
Hints
Hint 1
Index the correct log-probability at each position, negate, then average over the masked entries only.
Keywords
research-paperstransformersllmpretrainingbert
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.