Negative Sampling: Subsampling Frequent Words
Beginner
The problem, from first principles
The most frequent words, such as 'the', carry little information and dominate the training pairs. Word2vec discards some of their occurrences at random, which speeds up training and improves the quality of the rarer words.
From theory to code
Implement subsample_keep_prob(freq, t), returning the probability of keeping an occurrence, capped at 1.
Constraints
- The threshold
tis usually around 1e-5 to 1e-4.
Hints
Hint 1
Compute sqrt(t / freq) and take the minimum with 1.
Keywords
research-papersunsupervisedword-embeddingsnegative-sampling
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.