BERT: Summing Token, Segment and Position Embeddings
Beginner
The problem, from first principles
BERT's input must tell the model three things at once: which word it is, which sentence segment it belongs to, and where it sits in the sequence. The paper gives each its own learned table, and the three vectors are added together.
From theory to code
Implement bert_input_embedding, which looks up token, segment and position vectors and returns their element-wise sum for each position.
Constraints
- Use the first
Trows of the position table.
Hints
Hint 1
Index each table with the matching ids, slice the position table to length T, and add the three arrays.
Keywords
research-paperstransformersllmpretrainingbert
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.