Repetition & Diversity Metrics

Beginner NVIDIA, OpenAI+6

The problem, from first principles

Tuning decoding settings needs numbers, not impressions. Two simple metrics describe a generated text. Distinct-n is the fraction of its n-grams that are unique: a text that repeats itself has a low value, and a text with no repeated n-gram has distinct-n equal to 1. The repetition rate is the fraction of n-gram positions that repeat an earlier one, which is simply 1 - distinct-n for one text. For a set of samples from the same prompt, distinct-n computed over all samples pooled together measures diversity across samples: low values mean the model keeps writing the same thing every time.

From theory to code

Implement distinct_n and corpus_distinct_n.

Constraints

tokens is a list of tokens. distinct_n(tokens, n) is unique n-grams / total n-grams, or 0.0 if there are no n-grams.
corpus_distinct_n(samples, n) pools the n-grams of every sample (n-grams never span two samples) and returns unique pooled n-grams / total pooled n-grams, or 0.0 if there are none.
Return Python floats.

Hints

Hint 1

Build n-grams as tuples with zip(*[tokens[i:] for i in range(n)]) or a slice comprehension.

Hint 2

Sets of tuples count the uniques.

Keywords

decodingevaluationdiversity

Python

Run your code to see results

No results yet

Run your code to check it against the examples, or submit to run every test.