# LLM-as-Judge

> Use a capable frontier model to score another model's outputs at scale — assigning quality ratings that would otherwise require human annotators.

- **Category**: Language Models
- **Subcategory**: Benchmarking
- **Canonical URL**: https://designpattern.fyi/patterns/llm-as-judge/

---

## Description
**Intent**: Scale quality evaluation beyond what human annotation throughput allows, using a frontier model as a proxy for human judgment on the same outputs.

**Context**: Human evaluation is slow, expensive, and doesn't scale to continuous integration pipelines. Automated metrics like ROUGE and BLEU miss quality dimensions like helpfulness, tone, and reasoning quality. An LLM judge bridges the gap — faster than humans, richer than n-gram overlap.

**Solution**: Define an evaluation rubric covering quality criteria (accuracy, helpfulness, conciseness, safety). Prompt a capable judge model (GPT-4, Claude 3 Opus) with the rubric, the original prompt, and the model's response. Request a score (1-5 or pass/fail) with a brief rationale. For comparative evaluation, use pairwise preference: show the judge two responses and ask which is better. Calibrate against human annotation on a known subset before relying on the judge.



## Use Cases
Continuous integration quality gates that run on every model checkpoint. A/B testing between model versions at scale. Evaluating open-ended generation quality where n-gram metrics fail. Post-deployment monitoring of quality drift.





## Trade-offs


### Advantages

- Scales to thousands of evaluations per hour — infeasible with human annotators

- Captures nuanced quality dimensions (reasoning, tone, helpfulness) that automated metrics miss

- Pairwise comparison format produces reliable relative rankings




### Considerations & Drawbacks

- Judge model has systematic biases — positional bias (favors first response), verbosity bias, self-preference

- Circular evaluation — using one model to evaluate another doesn't catch their shared failure modes

- Judge quality degrades on tasks outside its own capability ceiling