rag-system-design

Overview

RAG systems fail in ways that are hard to see: wrong chunks retrieved, context window overflows, hallucinations on retrieved content, stale knowledge base. This skill builds RAG systems with rigorous evaluation at every layer.

When to Use

• Before building any system that grounds LLM responses in external data
• When an existing RAG system is returning low-quality or inconsistent results
• Before selecting an embedding model or vector database

Process

Step 1: Define the retrieval task

What questions will users ask? What documents will answer them? What does "good retrieval" look like? Write 50+ question-answer pairs as your evaluation set before writing any code.

Step 2: Design the ingestion pipeline

• Source: what documents? In what format? How frequently updated?
• Chunking strategy: fixed-size vs semantic vs structural
• Chunk size: smaller chunks = more precise retrieval; larger = more context per chunk
• Overlap: prevent context from being cut at chunk boundaries
• Metadata: document source, date, section header — critical for filtering and attribution

Step 3: Choose and evaluate the embedding model

Test at least 2 embedding models on your retrieval evaluation set. Measure: retrieval accuracy (does the correct chunk appear in top-k?), latency, cost. Don't default to the most popular model.

Step 4: Build the vector store

• Choose an index type appropriate for scale (exact vs approximate)
• Store metadata alongside vectors
• Plan for updates and deletions (not all vector DBs handle this well)
• Test retrieval latency at your expected query volume

Step 5: Implement retrieval with evaluation

Measure:

• Recall@k: Does the correct document appear in the top k results?
• Precision@k: Of the top k results, how many are relevant?
• MRR: Mean Reciprocal Rank — how highly is the correct document ranked?

Target: Recall@5 > 0.80 on your evaluation set before moving to generation.

Step 6: Design the context assembly

How do you combine retrieved chunks into a prompt context?

• Order matters: LLMs attend more to beginning and end
• Deduplication: don't include the same content twice
• Relevance threshold: don't include low-score chunks just to fill context
• Citation tracking: which chunks supported which claims?