Switch Transformers: Expert Capacity
Beginner
The problem, from first principles
Experts run as fixed-size matrix multiplies, so each one can only take a set number of tokens per batch. Switch Transformers set that capacity from the average load times a slack factor, and tokens beyond it are dropped.
From theory to code
Implement expert_capacity(n_tokens, n_experts, capacity_factor), returning the integer capacity per expert.
Constraints
- Round up to a whole number of tokens.
Hints
Hint 1
Multiply the factor by the average load per expert, then take the ceiling.
Keywords
research-paperstransformersllmmoerouting
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.