THE TOKENIZERS UNKNOWN-ID TRAP

Difficulty: Hard Tags: Tokenization, LLM Pipelines, Data Integrity

Story

A serving pipeline maps tokens to integer IDs. Unknown tokens must map to a designated UNK ID, but a fallback mistakenly maps them to zero, which is also a valid token. Given a vocabulary and a token sequence, encode the tokens and report the unknown count.

The Math

The vocabulary maps each known token to its integer ID. Any token absent from the vocabulary maps to the supplied UNK ID. Return the encoded sequence and the number of unknown occurrences. Repeated unknown tokens count repeatedly.

Input Format

vocab (dict mapping token string to ID), unk_id (int), tokens (list of strings).

Output Format

(encoded, unknown_count).

Constraints

• 1 <= len(vocab), len(tokens) <= 10^5, tokens contain no whitespace, IDs are integers.
• Time limit: 1.0 second.

Example

Input

vocab = {hello: 0, world: 1, ml: 2}, unk_id = 99
tokens = [hello, x, world, x, ml]

Output

encoded = [0, 99, 1, 99, 2]
unknown_count = 2

Explanation: Both occurrences of x map to 99 and are counted separately.