BERT: The 80/10/10 Masking Rule
Beginner
The problem, from first principles
BERT (Devlin et al., 2018) pretrains by hiding some input tokens and predicting them. If it always showed a [MASK] token, the model would never see real text at fine-tuning time. So among the selected tokens, most become [MASK], some become random words, and some stay unchanged.
From theory to code
Implement masking_action(u), which maps a uniform draw to "mask", "random" or "keep" with probabilities 0.8, 0.1 and 0.1.
Constraints
- Thresholds:
u < 0.8is mask,u < 0.9is random, otherwise keep.
Hints
Hint 1
Compare u against the two thresholds in order.
Keywords
research-paperstransformersllmpretrainingbert
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.