rag-expert

RAG Expert

Retrieval-augmented generation answers from your corpus rather than from model memory. Almost every failure is a retrieval failure: the model cannot ground an answer in a passage it was never given.

Core Concepts

The Pipeline

ingest → chunk → embed → index
                              ↘
query → rewrite → retrieve → rerank → assemble context → generate → cite

Debug it in that order. When answers are wrong, look at what was retrieved before touching the prompt — the passage is usually missing, not misread.

Retrieval Quality Sets the Ceiling

Generation cannot exceed retrieval. Measure them separately: recall@k for retrieval, groundedness for generation. Conflating the two produces months of prompt tuning against a chunking problem.

Semantic Search Is Not Search

Embeddings capture similarity of meaning, which is exactly wrong for exact identifiers, error codes, product SKUs and rare terms. Vector-only retrieval reliably fails on ERR_4021 and on surnames. Hybrid retrieval — dense plus lexical — is the default, not an optimisation.

Chunking Is the Highest-Leverage Decision

The chunk is the unit of retrieval and the unit of context. Too small and it loses the meaning that makes it findable; too large and it dilutes the embedding and wastes budget.

Chunking

Split on structure first, size second. Markdown headings, HTML sections, legal articles and code functions are natural boundaries; a fixed character count is a fallback, not a strategy.

def chunk_markdown(doc: str, target: int = 900, overlap: int = 120) -> list[Chunk]:
    """Split on headings, then pack sections up to a target size."""
    sections = split_on_headings(doc)          # keeps the heading with its body
    chunks, buffer, heading_path = [], "", []

    for section in sections:
        if len(buffer) + len(section.text) > target and buffer:
            chunks.append(Chunk(text=buffer, heading_path=list(heading_path)))
            buffer = buffer[-overlap:]         # carry context across the seam
        heading_path = section.heading_path
        buffer += section.text

    if buffer.strip():
        chunks.append(Chunk(text=buffer, heading_path=list(heading_path)))
    return chunks