Perplexity (exp of loss), the standard LM evaluation metric

The problem, from first principles

[03-next-token-cross-entropy]'s loss, the average negative log-probability the model assigned to the true next token, is mathematically exactly what training minimizes, but a raw loss VALUE (something like 2.3) doesn't have an intuitive, human-readable interpretation on its own: is 2.3 good? Perplexity re-expresses that same number in a genuinely more interpretable form: exp(loss). The reason this particular transformation is the standard one: a model assigning EQUAL probability to every word in a V-word vocabulary (the worst reasonable baseline, pure uniform random guessing) has cross-entropy loss exactly ln(V), so its perplexity is exactly exp(ln(V)) = V. Perplexity can therefore be read directly as "the model is behaving as if it were choosing UNIFORMLY among this many words," a genuinely intuitive scale (lower is better, and the number itself has a concrete "effective vocabulary size" meaning) that a raw log-loss number doesn't offer on its own.

From theory to code

Implement perplexity(loss), exp(loss), and perplexity_from_logits(logits, token_ids), computing [03-next-token-cross-entropy]'s loss directly and converting it.

Constraints

• perplexity(loss) = exp(loss), nothing more.
• loss is assumed to already be the MEAN-reduced Cross-Entropy loss (matching [03-next-token-cross-entropy]'s default reduction="mean" behavior), not a sum over many positions.
• Perplexity is always >= 1 for a nonnegative loss (exp of a nonnegative number is always >= 1), with 1 representing a PERFECT model (zero loss, full confidence correctly placed on every true next token).

Hints