Distilling the Knowledge: The Combined Loss
Beginner
The problem, from first principles
Distillation usually trains the student on both the true label and the teacher's soft targets. A mixing weight alpha sets how much to trust the teacher relative to the labels.
From theory to code
Implement combined_loss(hard_ce, soft_ce, alpha), returning alpha * soft_ce + (1 - alpha) * hard_ce.
Constraints
alphais between 0 and 1.
Hints
Hint 1
It is a convex combination of the two losses.
Keywords
research-papersclassical-mldistillationknowledge
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.