# Eval as Contract

> Treat your eval suite as a binding contract — releases ship only if evals pass, and changing evals is an architectural review, not a config tweak.

- **Category**: Agentic AI
- **Subcategory**: Governance & Observability
- **Canonical URL**: https://designpattern.fyi/patterns/eval_as_contract/

---

## Description
**Intent**: Treat the eval suite as the contract the agent must satisfy — releases ship only if evals pass.
**Context**: You ship an agent to real users and are expected to hold a stable quality bar release after release. You already have an eval suite that gives you a numeric read on quality. The problem is it's aspirational — engineers can ship past failing evals with enough justification. Stakeholders need that bar to be *enforced*, not just measured.
**Solution**: Define a tiered eval suite: **blocking evals** (must pass for release) and **advisory evals** (tracked but not blocking). Wire blocking evals into CI. Block PRs and releases when blocking evals fail. Treat eval *changes* as architectural changes — require review and signoff, not just a commit.



## Use Cases
- An eval suite exists that can be tiered into blocking and advisory.
- CI can be wired so blocking eval failures actually prevent release.
- The team is willing to treat eval changes as architectural changes (review + signoff).






## Trade-offs


### Advantages

- Quality bar is enforced, not aspirational — the gate is real.

- The eval suite earns its seat by being load-bearing infrastructure.




### Considerations & Drawbacks

- Bad or miscalibrated evals block legitimate releases — eval quality matters as much as agent quality.

- Calibration is an ongoing empirical effort, not a one-time setup.







---
**Reference**: [Original Source](https://www.agentpatternscatalog.org/patterns/eval-as-contract/)