bpe-single-merge — Netflix case

Netflix-inspired text-ranking pipeline is building a compact subword vocabulary from frequent token pairs. You need to perform one BPE merge correctly so the vocabulary-building process can continue.

Perform one BPE merge: scan tokens from left to right and replace every occurrence of the adjacent pair (a, b) by the single token merged. Matches do not overlap; the leftmost one is merged first.

Implement solve(tokens,a,b,merged).

Returns. Return a new list of tokens. If the pair does not occur the tokens are returned unchanged (as a new list).

Examples

Example 1

Input:

solve(['l', 'o', 'w', 'l', 'o', 'w'], 'l', 'o', 'lo')

Output:

['lo', 'w', 'lo', 'w']

Example 2

Input:

solve(['a', 'a', 'a'], 'a', 'a', 'aa')

Output:

['aa', 'a']