What does low-bit quantization save and sacrifice?

The bottleneck

Low-bit weight codes reduce storage and read traffic, but quantization errors can change behavior. Weight, activation and KV quantization target different tensors; one “4-bit” label does not describe the entire inference process.

The mechanism

Grouped quantization stores a scale and sometimes a zero point per group. Codes are rounded and clipped, then consumed by low-bit kernels or dequantized. AWQ uses activation information to identify sensitive weights and adjust scaling, controlling losses in low-bit weight quantization.

q = clip(round(w/s)+z);  ŵ = s(q−z)

Examples and evidence

For 10B weights, payload alone is about 20GB in FP16 and 5GB at 4 bits (decimal). Add scales, zero points and unquantized tensors. Fourfold weight reduction does not imply fourfold reduction in total memory or latency.

New constraints

Outliers, calibration data and task distributions affect error. Efficient kernel support determines performance. Validate long contexts, domain tasks and output behavior rather than relying on one average score.