Group Normalization

The problem, from first principles

Group Normalization divides channels into groups and normalizes within each group. Unlike BatchNorm which normalizes across batch dimension, GroupNorm normalizes within spatial dimensions and groups.

Where mu_g and sigma_g are computed per group (across spatial/channel within group).

From theory to code

Implement:

group_norm(x, num_groups, gamma, beta, eps=1e-5)

Constraints

• x shape: (N, C, H, W) or (N, C)
• num_groups divides C.
• gamma, beta same shape as x[0] (channels).
• Return normalized x, same shape.