GPipe: The Pipeline Bubble
Intermediate
The problem, from first principles
GPipe (Huang et al., 2019) splits a model into pipeline stages on different devices and feeds each step as micro-batches. At the start and end, some stages wait for work. That idle time is the bubble.
From theory to code
Implement pipeline_bubble_fraction(stages, micro), returning the idle fraction of a pipeline step.
Constraints
- Return a float.
Hints
Hint 1
The schedule has M + S - 1 time slots per stage, of which S - 1 are idle on each device; divide.
Keywords
research-paperssystemsparallelismpipeline
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.