haystack

Haystack

This skill is for building intelligent search and retrieval systems with Haystack, combining document processing, semantic search, and LLM integration.

Overview

This skill provides guidance for working with Haystack 2.0+, a modern framework for building production-grade RAG and search applications. Haystack enables building systems that combine:

• Document processing and indexing
• Semantic and hybrid search capabilities
• LLM integration for question-answering and reasoning
• Multi-stage pipelines for complex workflows
• Vector and keyword-based retrieval strategies

When to use

Use this skill when the user is working on:

• Building RAG Systems: Creating retrieval-augmented generation pipelines that combine document search with LLM reasoning
• Document Search & Indexing: Implementing semantic or hybrid search over document collections
• Question-Answering Systems: Building QA systems that retrieve relevant context and answer questions
• LLM Integration: Connecting language models with retrieval systems for grounded responses
• Pipeline Development: Creating multi-stage processing workflows with Haystack components
• Document Processing: Preparing, chunking, and indexing documents for retrieval
• Vector Store Setup: Configuring and managing document embeddings and vector databases

Core Concepts

Key Haystack Components

• DocumentStore: Storage backends for documents (ElasticsearchDocumentStore, InMemoryDocumentStore, WeaviateDocumentStore, etc.)
• Retriever: Components that fetch relevant documents (BM25Retriever, EmbeddingRetriever, HybridRetriever)
• Pipeline: DAG-based orchestration of components
• Generators/Answerers: LLM-powered components that generate responses
• Preprocessors: Components for text chunking, cleaning, and normalization

Architecture Patterns

1. Simple RAG Pipeline

Query → Retriever → LLM Generator → Answer

2. Hybrid Search Pipeline

Query → BM25 Retriever ⟶  Embedding Retriever ⟶ Document Joiner → LLM Generator → Answer