rag

Build a Railtracks RAG Pipeline

The user wants to build a retrieval-augmented generation (RAG) pipeline using the railtracks framework: $ARGUMENTS

How railtracks retrieval works

The pipeline has two paths:

Write path (ingestion): Loader → Chunker → Embedder → VectorStore Read path (retrieval): query → Embedder → VectorStore.search → RetrievalResult

RetrievalRuntime orchestrates both paths. It takes a chunker, embedder, and store at construction time, then exposes ingest_all() / ingest() for writes and retrieve() for reads.

Core data flow

Stage	Input	Output
Loader	source (file, URL, dataset)	Document
Chunker	Document	Chunk
Embedder	Chunk.content	EmbeddedChunk
Store	EmbeddedChunk	stored StoreEntry
Retrieve	query string	RetrievalResult (ranked RetrievedChunk list)

---

Component reference

Loaders

from railtracks.retrieval.loaders import (
    TextLoader,  # .txt / .md files
    CSVLoader,  # rows → documents; content_columns, id_column, ignore_columns
    JSONLoader,  # .json / .jsonl files
    PyPDFLoader,  # PDF file or folder; breakdown_strategy="page" (default), "paragraph", or "document"
    HuggingFaceDatasetLoader,  # HF Hub datasets; pass dataset_name, split, content_columns
    LangChainLoaderAdapter,  # wrap any LangChain document loader
)

Chunkers

from railtracks.retrieval.chunking import (
    RecursiveCharacterChunker,  # general text; chunk_size (chars), overlap
    MarkdownHeaderChunker,  # heading-aware splits for .md content
    SentenceChunker,  # sentence windows; chunk_size (# sentences), overlap
    SemanticChunker,  # embedding-driven breakpoints; variable chunk size
    FixedTokenChunker,  # hard token budget; chunk_size (tokens), overlap
)

Embedders

from railtracks.retrieval.embedding import (
    OpenAIEmbedding,  # default model: "text-embedding-3-small"
    AzureEmbedding,  # Azure OpenAI routing
    OllamaEmbedding,  # local dev; model, api_base
    LiteLLMEmbedding,  # any LiteLLM-supported provider
)