rag-pipeline-builder

IBM RAG Pipeline Builder

Purpose

Expert guidance for designing and implementing complete IBM RAG (Retrieval-Augmented Generation) pipelines using IBM Cloud services. Covers every stage from document ingestion to LLM answer generation.

IBM Cloud Product Coverage

IBM Cloud Product	RAG Stage
IBM Cloud Object Storage	Document source
IBM watsonx.ai	Embedding generation + LLM generation (Granite, Llama)
IBM watsonx.data (Milvus)	Primary vector store for high-scale RAG
IBM watsonx.data (OpenSearch)	Vector + BM25 hybrid search
IBM DataStax Astra DB (HCD)	Alternative vector store for global distribution

Rules

• Preferred embedding model: ibm/slate-125m-english-rtrvr (dim=768)
• Preferred generation models: ibm/granite-3-8b-instruct, meta-llama/llama-3-3-70b-instruct
• Use ibm_watsonx_ai.foundation_models.Embeddings for embedding generation
• Use langchain_ibm.WatsonxLLM or ibm_watsonx_ai inference API for generation
• Chunking default: RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=128)
• Hybrid search: always combine vector + BM25 (weight: 70% vector, 30% BM25)
• Reranking: use ibm/slate-125m-english-rtrvr cross-encoder or BM25 score fusion

---

Scope

• RAG architecture selection (Milvus vs OpenSearch vs AstraDB)
• Chunking strategy optimisation for different document types
• IBM watsonx.ai embedding and generation integration
• Hybrid search design and score normalisation
• RAG evaluation with RAGAS metrics
• MCP server design for RAG tool exposure

---

RAG Architecture Decision Matrix

Criterion	Milvus	OpenSearch	AstraDB
Scale	Billion-scale	Million-scale	Global scale
Hybrid search	Manual	Native BM25	Manual
Deployment	IBM watsonx.data	IBM watsonx.data	IBM HCD SaaS
Best for	Large-scale RAG	Enterprise search + RAG	Global SaaS RAG

---

Procedure

Phase 1: Embedding Generation

from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import Embeddings

embedder = Embeddings(
    model_id="ibm/slate-125m-english-rtrvr",
    credentials=Credentials(url="https://us-south.ml.cloud.ibm.com", api_key=IBM_API_KEY),
    project_id=WATSONX_PROJECT_ID,
)
query_vector = embedder.embed_query(user_question)
doc_vectors = embedder.embed_documents(chunks)