Group-Wise INT4 Weight Quantization (GPTQ/AWQ-style)

The problem, from first principles

INT4 has only 16 representable levels (-7..7 signed, using symmetric quantization), so a single scale per row ([02-per-channel-weight-quantization]'s approach) is often too coarse: any local variation in magnitude within a row gets quantized very lossily. Implement group-wise ("group quantization") INT4 weight quantization: split each row of a weight matrix into fixed-size contiguous groups along the input dimension, and quantize each group with its own independent scale — the scheme used by popular INT4 weight-only quantization methods (GPTQ, AWQ) in production LLM serving.

From theory to code

scale[o, g] = max(|W[o, g*group_size:(g+1)*group_size]|) / 7
Q[o, i] = clip(round(W[o, i] / scale[o, i // group_size]), -7, 7)

Constraints

• in_features must be divisible by group_size.
• Each contiguous group of group_size input features (per row) gets its own scale, computed the same way as symmetric quantization restricted to that group.
• INT4 signed symmetric range is [-7, 7].
• Return the quantized matrix, the (out_features, n_groups) scale matrix, and the dequantized reconstruction.

Hints