document-processing-for-rag

Document Processing for RAG

Document processing transforms raw files into chunked, searchable documents for RAG pipelines.

Installation

pip install langchain-community langchain-text-splitters langchain-huggingface faiss-cpu
# Or for GPU support:
pip install faiss-gpu

# For hybrid search (BM25 + semantic):
pip install rank-bm25

Processing Pipeline (Tutorial 03)

Load -> Split -> Embed -> Index

Document Loaders

from langchain_community.document_loaders import (
    TextLoader,
    DirectoryLoader,
    PyPDFLoader
)

# Single text file
loader = TextLoader("document.txt")
docs = loader.load()

# Directory of files
# loader_cls defaults to UnstructuredFileLoader (requires 'unstructured' package)
# TextLoader is simpler and works for plain text files
loader = DirectoryLoader("docs/", glob="*.txt", loader_cls=TextLoader)
docs = loader.load()

# PDF files (requires: pip install pypdf)
loader = PyPDFLoader("document.pdf")
pages = loader.load()

Text Splitters

RecursiveCharacterTextSplitter (Recommended)

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Characters per chunk (default: 4000)
    chunk_overlap=200,    # Overlap between chunks (default: 200)
    length_function=len   # Default length function
)

chunks = splitter.split_documents(documents)  # 'documents' from loader.load()

Why overlap? Prevents context loss at chunk boundaries.

CharacterTextSplitter (Simple)

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=0,
    separator="\n\n"  # Split on paragraphs (default)
)

# Usage
chunks = splitter.split_documents(documents)
# Or for raw text:
# text_chunks = splitter.split_text(text)

Chunking Best Practices

Chunk Size Guidelines (in characters with length_function=len):

• Small (200-500): Precise retrieval, may lack context
• Medium (500-1000): Recommended - Balance of precision and context
• Large (1000-2000): More context, less precise