ZeRO: The Shard Size
Beginner
The problem, from first principles
When parameters are partitioned, each GPU stores a contiguous shard. Every shard should be the same size, so the largest shard is the total divided by the device count, rounded up.
From theory to code
Implement shard_size(total, N), the ceiling of total over N.
Constraints
- Uneven totals leave the last shard smaller in practice.
Hints
Hint 1
Use integer ceiling division.
Keywords
research-paperssystemstrainingmemory
Python
Run your code to see results
No results yet
Run your code to check it against the examples, or submit to run every test.