# Post-Training Quantization (PTQ)

> Reduce model weight precision from FP16 to INT8 or INT4 after training with no retraining — trading a small accuracy loss for large memory reduction and inference speedup.

- **Category**: Language Models
- **Subcategory**: Quantization
- **Canonical URL**: https://designpattern.fyi/patterns/post-training-quantization/

---

## Description
**Intent**: Cut inference memory footprint and increase throughput by lowering numerical precision without touching the training pipeline.

**Context**: A 7B parameter model at FP16 requires ~14GB of VRAM. INT8 halves that; INT4 quarters it — making models that were GPU-cluster-only deployable on a single consumer GPU. No retraining needed — this is a post-hoc transformation on any existing checkpoint.

**Solution**: After training, quantize weights (and optionally activations) from FP16/BF16 to INT8 or INT4 using calibration data to determine per-layer scaling factors that minimize quantization error. Libraries: bitsandbytes (INT8/INT4), GPTQ (INT4 weight quantization), llama.cpp (GGUF). Run calibration on a representative dataset — random calibration data degrades quality.



## Use Cases
Deploying large models on memory-constrained hardware. Increasing inference throughput on a fixed GPU budget. Consumer and edge deployment of models too large for available VRAM at full precision.





## Trade-offs


### Advantages

- No retraining — applies to any existing checkpoint in minutes

- 2-4x memory reduction at INT8/INT4 with minimal quality regression at INT8

- Inference speedup on hardware with native INT8 support (most modern GPUs and NPUs)




### Considerations & Drawbacks

- Accuracy degrades — typically small at INT8, larger at INT4, varies significantly by model and task

- Some layers are more sensitive and may need to stay at higher precision (mixed-precision PTQ)

- Calibration data quality affects quantization grid — poor calibration data → worse accuracy