MoCo: The InfoNCE Loss With a Queue
Advanced
The problem, from first principles
MoCo trains the query to pick its positive key out of a large queue of negative keys. The loss is InfoNCE: a softmax over normalized similarities, scaled by a temperature.
From theory to code
Implement info_nce(q, k_pos, negatives, tau), normalizing the query and keys, then computing the softmax loss of the positive.
Constraints
- Normalize every vector before computing similarities.
Hints
Hint 1
Normalize the query and all keys, compute scaled dot products, then use logsumexp minus the positive logit.
Keywords
research-papersunsupervisedcontrastivemomentum
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.