Distilling the Knowledge: The Soft-Target Loss

Intermediate

The problem, from first principles

The student is trained to match the teacher's softened probabilities. Cross-entropy against those soft targets carries the teacher's view of which wrong classes are close, which the hard label cannot show.

From theory to code

Implement soft_target_ce(teacher_logits, student_logits, T), computing -sum(p_teacher * log q_student) * T**2.

Constraints

Both distributions use temperature T.

Hints

Hint 1

Build both softened distributions with a stable softmax, take the log of the student's, then sum the products and multiply by T**2.

Keywords

research-papersclassical-mldistillationknowledge

Python

Run your code to see results

No results yet

Run your code to check it against the examples, or submit to run every test.