# Parametric Memory vs. Retrieval-Augmented Generation (RAG)

> Should knowledge live in model weights (parametric) or be retrieved at inference time (RAG)? Knowledge freshness, latency, cost, hallucination risk, and scalability trade-offs.

- **Category**: Trade-offs
- **Subcategory**: LLMs
- **Canonical URL**: https://designpattern.fyi/trade_offs/parametric-vs-rag/

---

## Description
**Intent**: Choose between storing knowledge in model weights versus retrieving from external documents. Trade-offs involve knowledge freshness, latency, cost, hallucination risk, and scalability.

**Context**: Parametric memory (knowledge in model weights) has frozen knowledge at training cutoff but low latency. RAG retrieves current knowledge from documents but adds retrieval latency. Parametric has higher hallucination risk. RAG scales to billions of documents and provides citations. RAG enables privacy by keeping sensitive documents in private stores.

**Solution**: Default to RAG for enterprise/domain-specific applications. Use hybrid retrieval: sparse (BM25/TF-IDF) + dense (embedding similarity) with score fusion. Use re-ranking (cross-encoder) for improved precision. Optimize chunk size: 128-256 tokens for precision, 512-1024 for coherence. Monitor retrieval recall@K as separate metric.



## Use Cases
Enterprise knowledge base using RAG to provide current, citable answers. General Q&A using parametric knowledge for low latency. Medical diagnosis using RAG with up-to-date research papers.



## Implementation Example

```python
# Parametric vs. RAG: Hybrid Approach

from sentence_transformers import SentenceTransformer
import faiss
from sklearn.feature_extraction.text import TfidfVectorizer

class HybridRAG:
    def __init__(self, documents):
        self.documents = documents
        self.embedder = SentenceTransformer('all-MiniLM-L6-v2')

        # Dense retrieval (embeddings)
        self.dense_embeddings = self.embedder.encode(documents)
        self.dense_index = faiss.IndexFlatL2(self.dense_embeddings.shape[1])
        self.dense_index.add(self.dense_embeddings)

        # Sparse retrieval (BM25/TF-IDF)
        self.sparse_vectorizer = TfidfVectorizer()
        self.sparse_matrix = self.sparse_vectorizer.fit_transform(documents)

    def hybrid_retrieve(self, query, k=5, alpha=0.5):
        """Hybrid retrieval: dense + sparse with score fusion"""
        # Dense retrieval
        query_embedding = self.embedder.encode([query])
        dense_distances, dense_indices = self.dense_index.search(query_embedding, k * 2)

        # Sparse retrieval
        query_sparse = self.sparse_vectorizer.transform([query])
        sparse_distances, sparse_indices = self.sparse_search(query_sparse, k * 2)

        # Score fusion
        scores = {}
        for i, idx in enumerate(dense_indices[0]):
            scores[idx] = scores.get(idx, 0) + alpha * (1 - dense_distances[0][i])

        for i, idx in enumerate(sparse_indices):
            scores[idx] = scores.get(idx, 0) + (1 - alpha) * (1 - sparse_distances[i])

        # Get top-k
        top_k = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:k]
        return [self.documents[idx] for idx, score in top_k]

    def rerank(self, query, retrieved_docs, reranker_model):
        """Cross-encoder re-ranking for improved precision"""
        pairs = [[query, doc] for doc in retrieved_docs]
        scores = reranker_model.predict(pairs)
        sorted_docs = [doc for _, doc in sorted(zip(scores, retrieved_docs), reverse=True)]
        return sorted_docs

# Usage
rag_system = HybridRAG(document_corpus)
relevant_docs = rag_system.hybrid_retrieve(user_query, k=10)
reranked_docs = rag_system.rerank(user_query, relevant_docs, reranker_model)

```



## Trade-offs


### Advantages

- RAG provides current knowledge vs. frozen training data

- RAG reduces hallucination risk with source citations

- RAG scales to billions of documents

- Parametric provides lowest latency for general knowledge




### Considerations & Drawbacks

- RAG adds retrieval latency (50-500ms)

- RAG requires document management infrastructure

- Parametric knowledge can become stale

- RAG quality depends on retrieval system performance