Note: torch.nn.DataParallel vs DistributedDataParallel, What the Real APIs Do Differently

The problem, from first principles

01-data-parallelism-gradient-averaging implemented the idea of data parallelism, but PyTorch actually ships two different APIs for it — torch.nn.DataParallel (DP) and torch.nn.parallel.DistributedDataParallel (DDP) — and PyTorch's own documentation is explicit that DDP should always be preferred. This question quantifies exactly why, in terms of real communication volume and memory imbalance, rather than just citing the recommendation.

From theory to code

Implement dp_communication_volume, ddp_communication_volume, dp_gpu0_memory_multiplier, ddp_gpu0_memory_multiplier, and communication_reduction_factor, quantifying the two APIs' genuinely different communication and memory-imbalance costs.

Constraints

• dp_communication_volume(model_size_bytes, num_gpus) returns 2 * (num_gpus - 1) * model_size_bytes (0.0 when num_gpus <= 1) — DP re-broadcasts the model and gathers outputs to GPU0 on every single forward pass.
• ddp_communication_volume(model_size_bytes, num_gpus) returns 2 * (num_gpus - 1) / num_gpus * model_size_bytes (0.0 when num_gpus <= 1) — the standard ring all-reduce volume for one gradient sync.
• dp_gpu0_memory_multiplier(num_gpus) returns float(num_gpus); ddp_gpu0_memory_multiplier(num_gpus) returns 1.0 regardless of num_gpus.
• communication_reduction_factor returns dp_communication_volume / ddp_communication_volume (1.0 when num_gpus <= 1), and must depend only on num_gpus, not on the actual model_size_bytes passed in.

Hints

Open one at a time. Each gives away a little more than the last.