CLIP: The Symmetric Contrastive Loss
Advanced
The problem, from first principles
CLIP's loss treats each pair in both directions: for each image, pick its caption among the batch, and for each caption, pick its image. Averaging the two cross-entropies makes the objective symmetric.
From theory to code
Implement clip_symmetric_loss(logits), the mean of the row and column cross-entropies with diagonal targets.
Constraints
- Use
logsumexpfor stability.
Hints
Hint 1
For each direction, the loss per row is logsumexp(row) - diagonal entry. Average over rows for both the matrix and its transpose, then average the two.
Keywords
research-paperscomputer-visionvision-languagecontrastive
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.