# QLoRA (Quantized Low-Rank Adaptation)

> Fine-tune a 4-bit quantized base model using LoRA adapters in BF16 — enabling 65B parameter model fine-tuning on a single 48GB GPU.

- **Category**: Language Models
- **Subcategory**: Fine-Tuning
- **Canonical URL**: https://designpattern.fyi/patterns/qlora/

---

## Description
**Intent**: Combine 4-bit quantization's memory savings with LoRA's parameter efficiency — making fine-tuning of very large models possible on hardware that previously couldn't hold them even for inference.

**Context**: Standard LoRA still requires the base model in FP16 — a 65B model needs ~130GB VRAM just for frozen base weights. QLoRA quantizes the frozen base to 4-bit NF4 while LoRA adapters are trained in BF16, dequantizing on the fly for each forward pass.

**Solution**: Quantize the frozen base model to 4-bit Normal Float (NF4) using bitsandbytes. Attach LoRA adapters in BF16 to target layers. During training: dequantize the NF4 weight to BF16 for each forward pass, compute gradients in BF16, update only the LoRA adapter parameters. Apply double quantization (quantize the quantization constants themselves) and paged optimizers for additional memory savings on gradient spikes.



## Use Cases
Fine-tuning 13B, 33B, 65B, or 70B models on a single high-end GPU. Research and fine-tuning experiments with single-node or consumer GPU budgets. Any scenario where standard LoRA fits but the full FP16 base model does not.





## Trade-offs


### Advantages

- Makes 65B+ model fine-tuning accessible on a single 48GB GPU

- Accuracy close to full BF16 LoRA fine-tuning despite 4-bit base weights

- Paged optimizers handle memory spikes from gradient accumulation




### Considerations & Drawbacks

- Slower training than BF16 LoRA due to per-pass NF4 dequantization overhead

- NF4 base model has slightly lower quality floor than FP16 baseline

- More complex setup — requires bitsandbytes and careful memory budgeting