# Model Size vs. Inference Cost (Scaling Laws)

> Larger models are more capable but more expensive to serve. Chinchilla scaling laws, quantization, speculative decoding, knowledge distillation, and MoE architectures.

- **Category**: Trade-offs
- **Subcategory**: LLMs
- **Canonical URL**: https://designpattern.fyi/trade_offs/model-size-vs-inference-cost/

---

## Description
**Intent**: Balance model capability against inference cost. Larger models are more capable but exponentially more expensive to serve. Optimization techniques can reduce costs while maintaining quality.

**Context**: Chinchilla scaling laws show optimal compute efficiency when scaling model size and training tokens equally. Larger models require more GPU memory and compute for inference. Techniques like quantization, speculative decoding, knowledge distillation, and mixture-of-experts (MoE) can reduce serving costs while maintaining capability.

**Solution**: Prefer small, well-trained models over large undertrained models. Use INT8 quantization as default (2× memory reduction, minimal quality loss). Use AWQ over GPTQ for INT4. Use speculative decoding for latency-sensitive serving. Deploy MoE for capacity without proportional cost. Profile per-token serving cost before scaling.



## Use Cases
Production deployment using Llama-3-8B-Instruct for cost efficiency with fine-tuning. Latency-sensitive application using speculative decoding. Cost-optimized deployment using INT4 quantization with AWQ.



## Implementation Example

```python
# Model Size vs. Inference Cost Optimization

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from accelerate import infer_auto_device_map, dispatch_model

class OptimizedInference:
    def __init__(self, model_name, optimization_level='int8'):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = self.load_model(model_name, optimization_level)

    def load_model(self, model_name, optimization_level):
        if optimization_level == 'int8':
            # INT8 Quantization - 2x memory reduction
            model = AutoModelForCausalLM.from_pretrained(
                model_name,
                load_in_8bit=True,
                device_map="auto"
            )
        elif optimization_level == 'int4':
            # INT4 Quantization - 4x memory reduction
            model = AutoModelForCausalLM.from_pretrained(
                model_name,
                load_in_4bit=True,
                device_map="auto"
            )
        elif optimization_level == 'speculative':
            # Speculative decoding setup
            model = self.setup_speculative_decoding(model_name)
        else:
            # Baseline FP16/BF16
            model = AutoModelForCausalLM.from_pretrained(
                model_name,
                torch_dtype=torch.bfloat16,
                device_map="auto"
            )
        return model

    def setup_speculative_decoding(self, model_name):
        """Setup speculative decoding with draft model"""
        main_model = AutoModelForCausalLM.from_pretrained(
            model_name, torch_dtype=torch.bfloat16, device_map="auto"
        )
        draft_model = AutoModelForCausalLM.from_pretrained(
            "gpt2", torch_dtype=torch.bfloat16, device_map="auto"
        )
        return SpeculativeDecodingModel(main_model, draft_model)

    def estimate_cost(self, num_tokens, model_size_in billions):
        """Estimate inference cost based on model size"""
        # Simplified cost model
        cost_per_1k_tokens = model_size_in_billions * 0.001
        return (num_tokens / 1000) * cost_per_1k_tokens

    def profile_inference(self, prompt, max_tokens=100):
        """Profile latency and throughput"""
        import time
        start = time.time()
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        outputs = self.model.generate(**inputs, max_new_tokens=max_tokens)
        latency = time.time() - start
        tokens_generated = len(outputs[0]) - len(inputs[0])
        throughput = tokens_generated / latency
        return {"latency": latency, "throughput": throughput}

```



## Trade-offs


### Advantages

- Systematic approach to cost-optimized deployment

- Multiple techniques for different optimization goals

- Quantization provides significant cost savings with minimal quality loss

- Speculative decoding reduces latency without quality loss




### Considerations & Drawbacks

- Quantization sensitivity varies by task and model

- Speculative decoding requires draft model infrastructure

- MoE increases memory bandwidth requirements

- Small models may not handle complex reasoning tasks