Label smoothing: softening one-hot targets before cross-entropy

The problem, from first principles

A standard one-hot classification target ([1, 0, 0] for class 0 out of 3 classes) tells cross-entropy loss to push the model toward predicting probability EXACTLY 1.0 for the correct class and EXACTLY 0.0 for every other class. But [01-classical-ml/02-classification/10-logsoftmax-nllloss]'s softmax can only ever approach 1.0 or 0.0 in the limit, as its raw logits grow toward +infinity or -infinity, it can never actually REACH them. A model trained against a strict one-hot target is therefore being pushed to make its logits increasingly extreme forever, which tends to make the model dangerously OVERCONFIDENT: it starts assigning probability 0.9999 to predictions that are frequently wrong, especially on inputs that are genuinely ambiguous or mislabeled, rather than reporting a more honest, moderate confidence.

Label smoothing fixes this by simply refusing to ask for a perfect 1.0/0.0 target in the first place: instead of [1, 0, 0], the target becomes something like [0.9333, 0.0333, 0.0333], still clearly favoring the correct class, but leaving a small amount of probability mass spread across every OTHER class too. The model can now actually achieve its target loss of zero without needing infinitely large logits, which empirically makes trained models noticeably better calibrated (their stated confidence more closely matches their actual accuracy) and, in many cases, modestly improves accuracy itself.

From theory to code

Implement smooth_labels(one_hot, smoothing, num_classes). Given a one-hot target vector (or a batch of them), return the smoothed version: the true class's entry becomes 1 - smoothing, and EVERY class (including the true one) additionally gets smoothing / num_classes added on top.

Constraints

• The smoothed vector for a single example must still sum to exactly 1.0 (it remains a valid probability distribution).
• With smoothing=0, the output must exactly equal the original one-hot input (no smoothing at all).
• Every class that was originally 0 in the one-hot vector ends up at exactly smoothing / num_classes, not 0.
• The true class ends up at (1 - smoothing) + smoothing / num_classes, not exactly 1 - smoothing (the small uniform amount gets added to EVERY class, the true class included).

Hints