rag-hybrid-search

rag-hybrid-search

Implement hybrid search combining semantic vector retrieval with keyword-based BM25 search for improved RAG pipeline accuracy and recall.

Overview

Hybrid search addresses the limitations of pure semantic or pure keyword search:

• Semantic search excels at conceptual similarity but may miss exact matches
• Keyword search finds exact terms but lacks semantic understanding
• Hybrid combines both for superior retrieval performance

Capabilities

Search Strategies

• Dense vector semantic search (embeddings)
• Sparse vector keyword search (BM25, TF-IDF)
• Hybrid fusion with configurable weighting
• Reciprocal Rank Fusion (RRF) combination

Retrieval Configuration

• Configure embedding models for dense search
• Tune BM25 parameters (k1, b values)
• Set retrieval limits and thresholds
• Apply metadata filtering

Ranking & Reranking

• Score normalization across search types
• Weighted score fusion
• Cross-encoder reranking
• MMR (Maximum Marginal Relevance) diversity

Index Management

• Create and update hybrid indexes
• Batch indexing with progress tracking
• Index optimization and maintenance
• Multi-index federation

Usage

Basic Hybrid Search with LangChain

from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Chroma
from langchain.retrievers import EnsembleRetriever
from langchain_openai import OpenAIEmbeddings

# Create documents
docs = [...]  # Your document chunks

# Dense retriever (semantic)
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)
dense_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Sparse retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 5

# Hybrid ensemble
hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, dense_retriever],
    weights=[0.4, 0.6]  # Adjust based on use case
)

# Query
results = hybrid_retriever.invoke("How do I configure the system?")