# Weight-Only Quantization

> Store weights at INT4 but keep activations in FP16 — getting most of the memory win with a fraction of the accuracy cost of quantizing both.

- **Category**: Language Models
- **Subcategory**: Quantization
- **Canonical URL**: https://designpattern.fyi/patterns/weight-only-quantization/

---

## Description
**Intent**: Model weights dominate memory; activations dominate compute precision sensitivity. Quantizing only weights gets most of the memory reduction while keeping compute in FP16.

**Context**: Quantizing both weights and activations to INT4 is aggressive and hurts accuracy significantly. But most of a large model's memory is weights, not activations. Dequantizing weights to FP16 just-in-time for each matrix multiplication keeps arithmetic in FP16 while storage stays at INT4.

**Solution**: Store weight matrices in INT4 (or INT3). At each layer's forward pass, dequantize the weight matrix from INT4 to FP16, perform the matmul in FP16, then discard the dequantized copy. Activations remain in FP16 throughout the computation. GPTQ, AWQ, and GGUF all use this approach. Calibration determines the optimal quantization grid per weight matrix.



## Use Cases
Memory-constrained inference (consumer GPU, laptop, on-device) where model weights are the bottleneck. Running 70B+ models on hardware that can't hold them at FP16. Prioritizing accuracy over maximum arithmetic throughput on batch inference.





## Trade-offs


### Advantages

- Approximately 4x memory reduction over FP16 with minimal accuracy loss on most architectures

- Arithmetic remains in FP16 — avoids INT4 matmul precision issues entirely

- Widely supported — GPTQ, AWQ, GGUF are mature, well-maintained ecosystems




### Considerations & Drawbacks

- Dequantization overhead on every forward pass reduces raw throughput vs. native INT4 compute

- Slower than unquantized FP16 on batch inference where memory is not the bottleneck

- Per-matrix calibration required for accurate quantization grid selection