Stretch: Basic Knowledge Distillation (Reuses KL Divergence)
The problem, from first principles
01/02 shrink a model by removing weights after the fact. Knowledge distillation compresses knowledge a completely different way: train a small "student" model from scratch, but instead of only learning from the true labels, also train it to match a large, already-trained "teacher" model's full output distribution — which carries far more information than a single correct-answer label ever could (a teacher's confident "70% cat, 25% dog, 5% everything else" reveals real relationships between classes that a bare label "cat" throws away entirely).
From theory to code
Implement distillation_loss(student_logits, teacher_logits, true_labels, temperature=2.0, alpha=0.5), Hinton et al.'s original distillation loss: a weighted combination of matching the teacher's temperature-softened distribution (via 03-kl-divergence) and getting the true labels right (ordinary cross-entropy).
Constraints
student_logits,teacher_logits: shape(batch_size, num_classes).true_labels: shape(batch_size,), integer class indices.temperature: divides both logit sets before softmax, "softening" both distributions.- The soft-target loss is scaled by
temperature**2(part of the original published formula). alphaweights the soft loss;(1 - alpha)weights the hard-label loss.alpha=1.0is pure soft-target matching,alpha=0.0is pure ordinary cross-entropy.- Returns a single Python
float.
Hints
Open one at a time. Each gives away a little more than the last.
Hint 1
Dividing logits by temperature before softmax produces a "softer" (more spread-out, less confident-looking) distribution — a higher temperature reveals more of a confident model's relative confidence across every class, not just its single top prediction.
Hint 2
Compute the soft loss per sample (kl_divergence(teacher_soft[i], student_soft[i], base=np.e) for each row i), average across the batch, then multiply by temperature**2 — this rescaling isn't optional decoration, it's part of the published formula, correcting for how dividing logits by T also shrinks the soft loss's own gradient magnitude by 1/T^2.
Keywords
No results yet
Run your code to check it against the examples, or submit to run every test.