mcp-crawl4ai-rag: Instructions file for Claude Code

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is a Model Context Protocol (MCP) server that integrates Crawl4AI with multiple vector databases and AI providers for advanced web crawling and RAG (Retrieval Augmented Generation) capabilities. The server enables AI agents and coding assistants to crawl websites, store content in vector databases, and perform intelligent document retrieval with optional AI hallucination detection using Neo4j knowledge graphs.

Key Features:

• Multi-AI Provider Support: OpenAI, Ollama, vLLM, and hybrid configurations
• Multi-Modal RAG: Text, image, and vision model support via vLLM
• Multiple Vector Databases: Supabase, SQLite, Neo4j, Pinecone, Weaviate
• Flexible Deployment: Docker orchestration with various provider combinations
• Cost Optimization: Mix providers (e.g., vLLM embeddings + OpenAI LLM)
• Privacy Options: Fully local deployment with Ollama
• Enterprise Ready: Production deployment with monitoring and health checks

Development Commands

Docker (Recommended)

Quick Start with Provider Selection:

# OpenAI + Supabase (Production)
docker compose --profile supabase up -d

# Ollama + SQLite (Local Development)
docker compose --profile ollama-sqlite up -d

# Hybrid OpenAI/Ollama + Supabase (Cost-Optimized)
docker compose --profile hybrid up -d

# Full Ollama Stack with Neo4j and Monitoring
docker compose --profile ollama-full up -d

Custom Build:

# Build with specific providers
docker build -t mcp/crawl4ai-rag \
  --build-arg AI_PROVIDER=ollama \
  --build-arg VECTOR_DB_PROVIDER=sqlite .

# Run with environment file
docker run --env-file .env -p 8051:8051 mcp/crawl4ai-rag

Direct Python Development

# Install dependencies
uv pip install -e .
crawl4ai-setup

# Configure AI provider (examples)
export AI_PROVIDER=openai  # or ollama, vllm, mixed
export OPENAI_API_KEY=your_key

# OR for Ollama
export AI_PROVIDER=ollama
export OLLAMA_BASE_URL=http://localhost:11434

# OR for vLLM
export AI_PROVIDER=vllm
export VLLM_BASE_URL=https://your-vllm-endpoint.com/v1
export VLLM_API_KEY=your_vllm_api_key

# Run the MCP server
uv run src/crawl4ai_mcp.py