BIO Tagging & Entity Spans

The problem, from first principles

Fine-tuning a language model for named entity recognition attaches a classifier to every token. The labels use the BIO scheme: B-X marks the beginning of an entity of type X, I-X marks a token inside the same entity, and O marks a token outside any entity. The model outputs one tag per token, but what applications need is spans, such as ("PER", 0, 2) for the first two tokens being a person. Decoding is a small state machine, and it must cope with the malformed sequences real models produce, such as an I-X tag that does not follow a B-X.

From theory to code

Implement decode_bio.

Constraints

• tags is a list of strings such as 'B-PER', 'I-PER', 'O'. Return a list of (type, start, end) spans with end exclusive, in order.
• B-X always starts a new span. I-X extends the current span if one is open and has type X; otherwise (no open span, or a different type) it starts a new span of type X (lenient decoding). O closes any open span.
• A span that is still open at the end of the list is closed there.

Hints