Note: Model/Pipeline Parallelism — Why Frontier Training Needs It, and What It Costs

The problem, from first principles

01-data-parallelism-gradient-averaging assumes every worker can hold the WHOLE model — it just splits the batch. That assumption breaks for frontier-scale models that don't fit on a single GPU at all, which forces splitting the model itself instead: pipeline parallelism puts different LAYERS on different GPUs and streams "microbatches" through them like an assembly line — but an assembly line has a real, quantifiable cost: while it's filling up and draining, some stages are always sitting idle.

From theory to code

Implement pipeline_bubble_fraction(num_stages, num_microbatches), pipeline_wall_time, ideal_wall_time, and pipeline_stage_utilization, quantifying exactly how much of a pipeline's total runtime is wasted idle time ("bubble"), using the standard textbook pipeline-parallelism cost model.

Constraints

• pipeline_wall_time(p, m, t) returns (p - 1 + m) * t — the p - 1 steps needed to fill/drain the pipeline, plus m steady-state steps.
• ideal_wall_time(m, t) returns m * t — what a hypothetical zero-bubble pipeline would take.
• pipeline_bubble_fraction(p, m) returns (p - 1) / (p - 1 + m), algebraically equal to 1 - ideal_wall_time / pipeline_wall_time.
• pipeline_stage_utilization(p, m) returns 1.0 - pipeline_bubble_fraction(p, m).

Hints

Open one at a time. Each gives away a little more than the last.