# Prompt Variant Evaluation

> Author 2-N prompt variants, batch them against a frozen eval dataset, and let automated scoring pick the winner — prompt decisions become measurements, not taste.

- **Category**: Agentic AI
- **Subcategory**: Verification & Reflection
- **Canonical URL**: https://designpattern.fyi/patterns/prompt_variant_evaluation/

---

## Description
**Intent**: Replace 'which prompt feels better in the demo' with 'which prompt scores better on the eval set.'

**Context**: You're iterating on a prompt — different wording, different examples, different model bindings. Choosing between variants by demo or author taste produces non-reproducible decisions and loses the comparison the moment the demo is closed.

**Solution**: Build a prompt-flow harness with variant slots. For each slot, author 2-N variants. The harness runs all variants against a frozen eval dataset and rubric, scores them (deterministic checker, LLM-judge, or both), and surfaces per-variant scores plus per-item differences. Team picks the winner from the scores. This is offline and batched — distinct from shadow/canary testing on live traffic.


## Use Cases
- Multiple plausible prompt variants exist and the team needs to pick among them.
- An eval dataset and rubric exist (evaluation-driven development is in place).
- Inference cost permits batched comparison.





## Trade-offs


### Advantages

- Prompt decisions become measurements with an audit trail.

- Surfaces unexpected variant strengths the author would have missed in a demo.

- Composes with eval-driven development: variant evaluation is the unit of progress.




### Considerations & Drawbacks

- Running many variants multiplies inference cost.

- Variants can be tuned to game a weak rubric — the rubric must be honest.

- Authors over-iterate when every change is cheap to evaluate.







---
**Reference**: [Original Source](https://www.agentpatternscatalog.org/patterns/prompt-variant-evaluation/)