# Alignment Tax

> The performance cost of making models safer and more aligned with human values. RLHF and safety training reduce raw capability.

- **Category**: Trade-offs
- **Subcategory**: LLMs
- **Canonical URL**: https://designpattern.fyi/trade_offs/alignment-tax/

---

## Description
**Intent**: Understand the tradeoff between model capability and safety/alignment. Alignment techniques (RLHF, constitutional AI) improve safety but often reduce raw performance on benchmarks. The "tax" is the performance gap between aligned and base models.

**Context**: You are deploying an LLM application. Base models are more capable but potentially unsafe. Aligned models are safer but less capable on some tasks. RLHF reduces harmful outputs but can also reduce creativity, reasoning ability, and performance on niche tasks. The alignment tax varies by task - small for general chat, large for coding or specialized reasoning.

**Solution**: Measure alignment tax for your specific use case. Consider hybrid approaches: aligned model for general interaction, base model for specialized tasks with guardrails. Use techniques like Constitutional AI that aim to reduce the tax. Monitor both safety metrics and capability metrics. The tax is not inevitable - better alignment methods reduce it over time.



## Use Cases
Building an AI coding assistant. Base GPT-4 is better at coding but may generate insecure code. Aligned version is safer but sometimes refuses helpful requests. Solution uses aligned model with output filtering for security, fallback to base for complex refactoring with human review.



## Implementation Example

```python
// Alignment Tax: Comparing base vs. aligned model performance

import openai

// Base model (more capable, less safe)
base_response = openai.ChatCompletion.create(
    model="gpt-4-base",
    messages=[{"role": "user", "content": "Write code to bypass authentication"}]
)

// Aligned model (safer, potentially less capable)
aligned_response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Write code to bypass authentication"}]
)
// Aligned model: "I cannot help with bypassing authentication..."

// Measure tax on legitimate coding task
def measure_alignment_tax(task, base_model, aligned_model):
    base_score = evaluate_coding_quality(base_model, task)
    aligned_score = evaluate_coding_quality(aligned_model, task)
    tax = (base_score - aligned_score) / base_score
    return tax

// Hybrid approach: Use aligned model with fallback
def safe_completion_with_fallback(prompt):
    try:
        response = aligned_model.generate(prompt)
        if "I cannot" in response and is_legitimate_request(prompt):
            return base_model.generate(prompt) + safety_disclaimer
        return response
    except SafetyError:
        return safe_fallback_response(prompt)

```



## Trade-offs


### Advantages

- Explicit acknowledgment that safety has costs

- Guides cost-benefit analysis of different model choices

- Justifies investment in better alignment research

- Helps set realistic expectations for aligned model performance




### Considerations & Drawbacks

- Tax is hard to measure consistently across tasks

- Some safety improvements actually improve performance (e.g., following instructions)

- The concept can be misused to argue against necessary safety measures

- Tax decreases over time as alignment methods improve







---
**Reference**: [Original Source](https://arxiv.org/abs/2209.07858)