Masked-Token Corruption

The problem, from first principles

BERT learns language by a fill-in-the-blank game. Some tokens are hidden and the model must recover them using context from both sides. The corruption recipe has a subtle design. About 15% of the (non-special) tokens are chosen as targets. A chosen token is replaced by the [MASK] token only 80% of the time, by a random token 10% of the time and left unchanged 10% of the time. Why not always mask? Because [MASK] never appears in real text at fine-tuning time. Sometimes seeing a random or unchanged token forces the model to keep a good representation of every token, since any token might be a target. The labels are the original ids at chosen positions and an ignore value everywhere else.

From theory to code

Implement mask_tokens.

Constraints

• ids is a 1-D integer array. special_ids is a set of ids that may never be chosen. rng is a np.random.RandomState. Draw random numbers in exactly this order: (1) u = rng.random_sample(len(ids)); a position is chosen iff u < mask_prob and its id is not special. (2) r = rng.random_sample(num_chosen) for the chosen positions in increasing order. (3) rand_tokens = rng.randint(0, vocab_size, size=num_chosen).
• For chosen position number m: if r[m] < 0.8 the input becomes mask_id; elif r[m] < 0.9 it becomes rand_tokens[m]; else it is unchanged.
• Return (inputs, labels): inputs is a corrupted copy of ids, and labels equals the original id at chosen positions and -100 elsewhere.

Hints