CLIP: Zero-shot Classification
Beginner
The problem, from first principles
CLIP can classify images it was never trained on. Each class name is turned into a text prompt, and the image is assigned to the prompt it is most similar to. No labelled training data for the new classes is needed.
From theory to code
Implement zero_shot_predict(img_emb, class_embs), returning the index of the best-matching class prompt.
Constraints
- Returns a Python int.
Hints
Hint 1
Normalize the image and class embeddings, compute all the cosine similarities, then take the argmax.
Keywords
research-paperscomputer-visionvision-languagecontrastive
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.