# Byte-Pair Encoding (BPE) Tokenization

> Represent text as subword tokens learned from corpus frequency — balancing vocabulary coverage against sequence length, handling rare words by decomposing them into known pieces.

- **Category**: Language Models
- **Subcategory**: Tokenization
- **Canonical URL**: https://designpattern.fyi/patterns/bpe-tokenization/

---

## Description
**Intent**: Build a vocabulary of subword units that handles rare and unseen words without OOV failures — by decomposing them into known pieces rather than failing or using a generic unknown token.

**Context**: Word-level tokenization produces massive vocabularies and fails on rare words. Character-level handles everything but produces very long sequences that stress the context window. BPE finds the middle ground — common words become single tokens, rare words decompose into subword pieces.

**Solution**: Initialize with a character-level vocabulary. Iteratively merge the most frequent adjacent pair of tokens into a new compound token, adding it to the vocabulary. Repeat until the target vocabulary size is reached (typically 32K–100K). Byte-level BPE (GPT-2/GPT-4 tokenizer) starts from raw bytes — guaranteeing zero OOV on any Unicode input.



## Use Cases
Pre-training tokenizer design for any new model. Evaluating whether a standard tokenizer fragments domain-specific vocabulary (code, medical, legal) into inefficient pieces. Multilingual models where character and byte coverage matters.





## Trade-offs


### Advantages

- Handles rare and unseen words gracefully through subword decomposition — no OOV

- Vocabulary size is a tunable parameter — balance between sequence length and embedding table size

- Byte-level BPE eliminates OOV entirely — any Unicode input is encodable




### Considerations & Drawbacks

- Domain-specific terms may fragment into many tokens — inflating sequence length and wasting context

- Tokenization is model-specific — mismatching tokenizer to model corrupts input silently

- Vocabulary size trades off sequence efficiency against embedding table memory