gradient-accumulation — Cloudflare case

Cloudflare-inspired training job uses micro-batches because the full batch does not fit comfortably in memory. You need to accumulate and average gradients across the requested number of micro-batches before the optimizer step.

grads has one row per micro-batch (all rows have the same length). Return the element-wise average of the rows, the gradient a single large batch would have produced.

Implement solve(grads).

Returns. Return a float NumPy vector.

Examples

Example 1

Input:

solve([[1, 2], [3, 0], [-1, 1]])

Output:

[1.0, 1.0]

Example 2

Input:

solve([[2.0, 4.0]])

Output:

[2.0, 4.0]