# Red-Teaming

> Systematically probe the model for safety failures, jailbreaks, and harmful outputs using adversarial inputs — before users discover them in production.

- **Category**: Language Models
- **Subcategory**: Benchmarking
- **Canonical URL**: https://designpattern.fyi/patterns/red-teaming/

---

## Description
**Intent**: Find failure modes, safety vulnerabilities, and harmful output patterns before deployment — in structured testing, not in incident reports.

**Context**: Models trained to be helpful will produce harmful outputs when given adversarial inputs, unexpected edge cases, or sufficiently creative prompt sequences. Standard benchmark safety scores measure average-case behavior. Red-teaming probes the tail — the cases where failure has real consequences.

**Solution**: Assemble a red team (human adversaries, automated attack generation, or both). Define attack categories: jailbreaks (bypassing safety training), prompt injection (hijacking via malicious tool outputs or documents), harmful content elicitation, privacy extraction, misinformation generation, role-play escalation. For automated red-teaming, use a separate attacker LLM to generate adversarial prompts at scale. Document every failure with reproduction steps and severity rating.



## Use Cases
Pre-deployment safety evaluation for any user-facing model. Regression testing after fine-tuning updates that touch safety behavior. Testing multi-agent pipelines where prompt injection via tool outputs is a real attack vector.





## Trade-offs


### Advantages

- Finds real failure modes that standard benchmarks miss — tail behavior, not average behavior

- Adversarial attack patterns directly inform targeted safety fine-tuning

- Documents known risks with reproduction cases for compliance and responsible disclosure




### Considerations & Drawbacks

- Manual red-teaming doesn't scale — automated red-teaming requires a capable attack model

- Coverage is necessarily incomplete — you can only test attacks you think to try

- Model patches for discovered attacks can be bypassed by variants of the original