implementing-document-indexing

Document Indexing Implementation

Target: $ARGUMENTS

Implements a document indexing and hybrid retrieval pipeline: parse documents, build a heading-based tree index, chunk by heading boundaries, embed with sentence-transformers, store in FAISS, and retrieve via hybrid search.

Architecture Overview

Document --> Parser --> Pages --> TreeIndex (PageIndex)
                                     |
                                     v
                              Chunker (heading-boundary + max-token)
                                     |
                                     v
                              Embedder (sentence-transformers)
                                     |
                                     v
                              VectorStore (FAISS IndexFlatIP)
                                     |
                                     v
                         HybridRetrieval (vector search -> full page -> tree filter)

Chunking Strategy

See references/chunking-strategies.md for full reference.

Heading-boundary chunking (primary):

1. Split document by heading boundaries (H1-H6)
2. Each section becomes a chunk with heading hierarchy as metadata
3. If a section exceeds max tokens, split at sentence boundaries
4. Preserve heading path (e.g., H1 > H2 > H3) as chunk metadata

Max-token splits (fallback):

• Default max: 512 tokens
• Overlap: 64 tokens between splits
• Never split mid-sentence

Retrieval Pipeline

See references/retrieval-patterns.md for full reference.

Hybrid retrieval (vector search + tree filter):

1. Embed query with same model used for indexing
2. Vector search top-k chunks from FAISS (cosine similarity via IndexFlatIP)
3. Full page fetch -- retrieve complete pages containing matched chunks
4. Tree filter -- use PageIndex tree to filter to relevant sections only
5. Return filtered sections with source citations (page, heading path)

Data Models

@dataclass
class Document:
    pages: list[Page]
    metadata: dict[str, str]

@dataclass
class Page:
    number: int
    content: str
    headings: list[str]

@dataclass
class TreeNode:
    heading: str
    level: int
    content: str
    children: list[TreeNode]

    def filter(self, predicate: Callable) -> TreeNode | None: ...