Data filtering and contamination: keeping eval data out of the training set

The problem, from first principles

A model's evaluation score is only meaningful if the evaluation questions were GENUINELY unseen during training; if a benchmark's text (or something close enough to it) leaked into the training corpus, the model can "solve" it by partial memorization rather than by the actual capability the benchmark claims to measure, inflating reported scores in a way that doesn't reflect real ability. "Contamination" checking is exactly [04-deduplication]'s near-duplicate detection idea, applied ACROSS two different sets (training data vs. evaluation data) instead of WITHIN one set: the goal isn't merely "is this document unique," it's specifically "does this training document overlap with anything from the evaluation set that must stay unseen."

From theory to code

Implement get_ngrams(tokens, n) (every contiguous n-token window, as a set), has_contamination(train_doc_tokens, eval_ngrams, n) (does a training document share any n-gram with a precomputed evaluation-set n-gram pool), and filter_contaminated_documents(train_documents, eval_documents, n), removing every contaminated training document.

Constraints

• n-grams are built from CONTIGUOUS windows of exactly n tokens; a document shorter than n tokens produces an EMPTY n-gram set.
• The evaluation-set n-gram pool is built from EVERY evaluation document combined (a training document contaminating ANY one of them counts as contamination), not just the first.
• A single shared n-gram is sufficient to flag contamination (len(train_ngrams & eval_ngrams) > 0), no minimum overlap COUNT required.
• Larger n is a STRICTER, more specific contamination signal (fewer false positives from ordinary common phrasing); smaller n is looser and catches more, at the cost of more coincidental matches.

Hints

eval_ngrams = set()
for doc in eval_documents:
    eval_ngrams |= get_ngrams(doc, n)
return [i for i, doc in enumerate(train_documents) if not has_contamination(doc, eval_ngrams, n)]