Distilling the Knowledge: Softmax With Temperature
Beginner
The problem, from first principles
Hinton et al. (2015) observed that a large model's softmax is nearly one-hot, hiding how it ranks the wrong classes. Dividing the logits by a temperature T above 1 softens the distribution and exposes that ranking, which a smaller student can learn from.
From theory to code
Implement softmax_temperature(logits, T), returning the softmax of logits / T.
Constraints
Tis positive.
Hints
Hint 1
Divide by T, subtract the maximum for stability, exponentiate and normalize.
Keywords
research-papersclassical-mldistillationknowledge
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.