Repetition, Frequency & Presence Penalties

The problem, from first principles

Language models love to loop: once a phrase appears, its tokens become more likely to appear again, and greedy or low-temperature decoding can repeat the same sentence forever. Decoding libraries fight this by editing the logits of tokens that have already been generated. Three common rules exist. The repetition penalty (from the CTRL paper) divides the logit of a seen token by a factor when it is positive and multiplies it when it is negative, so the token always becomes less likely. The frequency penalty subtracts an amount proportional to how many times the token has appeared, and the presence penalty subtracts a flat amount once if it has appeared at all.

From theory to code

Implement apply_penalties.

Constraints

• logits is a 1-D float array over the vocabulary and generated_ids is a list of the token ids produced so far.
• First apply the repetition penalty rep > 0 to every distinct generated token: a positive logit becomes logit / rep, a negative logit becomes logit * rep (rep = 1.0 changes nothing).
• Then, for every token with count c > 0, subtract freq * c + pres.
• Tokens never generated are untouched. Return a new array and do not modify logits.

Hints