Negative Sampling: The Noise Distribution
Intermediate
The problem, from first principles
Negatives should be random words, but not only the most common ones, which would be trivial to reject. Word2vec draws negatives from the unigram distribution raised to the power 3/4, a compromise that gives rare words more chances than their raw frequency.
From theory to code
Implement unigram_noise(counts, power), returning the normalized counts raised to power.
Constraints
- The default power is 0.75.
Hints
Hint 1
Raise the counts to power, then divide by the sum.
Keywords
research-papersunsupervisedword-embeddingsnegative-sampling
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.