CLIP: The Similarity Matrix
Intermediate
The problem, from first principles
CLIP (Radford et al., 2021) trains an image encoder and a text encoder so that matching pairs have high similarity. The first step is the full matrix of similarities between every image and every caption in the batch.
From theory to code
Implement clip_logits(img, txt, temp), returning the scaled cosine similarity matrix.
Constraints
- Normalize each embedding before the dot products.
Hints
Hint 1
Divide each row by its norm, multiply the image matrix by the transposed text matrix, then divide by the temperature.
Keywords
research-paperscomputer-visionvision-languagecontrastive
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.