Negative Sampling: The Per-pair Loss
Advanced
The problem, from first principles
Negative sampling (Mikolov et al., 2013) replaces the full softmax with a binary task: is this word a real context, or one of a few random words? Each training pair then costs only one positive and a handful of negative dot products.
From theory to code
Implement negative_sampling_loss(v_c, u_o, U_neg), the logistic loss for the true pair plus the sampled negatives.
Constraints
- Use
logaddexpto compute-log sigma(x)stably.
Hints
Hint 1
The positive term is -log sigma(u_o . v_c); each negative term is -log sigma(-u_n . v_c). Sum all of them.
Keywords
research-papersunsupervisedword-embeddingsnegative-sampling
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.