RAGElo copilot-instructions.md

Copilot Instructions — RAGElo

Project Overview

RAGElo is a Python library and CLI for evaluating RAG (Retrieval-Augmented Generation) agents using Elo-based tournament ranking. It evaluates both document retrieval quality and answer quality across multiple RAG pipeline variations.

Commands

# Install for development
uv pip install -e '.[dev]'

# Run all tests (OpenAI integration tests are skipped by default)
pytest tests/

# Run with OpenAI integration tests (requires OPENAI_API_KEY)
pytest tests/ --runopenai

# Run a single test file or specific test
pytest tests/unit/test_experiment.py -v
pytest tests/unit/test_experiment.py::TestExperiment::test_method -v

# Lint and format (ruff rules: E, F, I; line-length: 119)
ruff check ragelo/
ruff format ragelo/

# Type checking (uses pydantic.mypy plugin)
mypy ragelo/

# Pre-commit hooks (ruff lint + format)
pre-commit run --all-files

Architecture

Factory + Registry Pattern

All major components use decorator-based factory registration. Enum types in ragelo/types/types.py define valid component names (RetrievalEvaluatorTypes, AnswerEvaluatorTypes, LLMProviderTypes, AgentRankerTypes).

# Registration (on class definition)
@RetrievalEvaluatorFactory.register(RetrievalEvaluatorTypes.REASONER)
class ReasonerEvaluator(BaseRetrievalEvaluator): ...

# Instantiation (via factory functions)
evaluator = get_retrieval_evaluator("reasoner", llm_provider=provider)
provider = get_llm_provider("openai", api_key="...")
ranker = get_agent_ranker("elo")

When adding a new evaluator, LLM provider, or ranker:

1. Add the name to the corresponding StrEnum in ragelo/types/types.py.
2. Create a config class in ragelo/types/configurations/.
3. Implement the class and decorate it with the factory's @register.
4. The class is now accessible via the factory function and CLI.

Evaluator Hierarchy

BaseEvaluator (ragelo/evaluators/base_evaluator.py) — async-first, abstract
├── BaseRetrievalEvaluator — evaluates document relevance (Query + Document → score)
│   Implementations: Reasoner, RDNAM, DomainExpert, FewShot, CustomPrompt
└── BaseAnswerEvaluator — evaluates answer quality (Query + AgentAnswer → score/winner)
    Implementations: Pairwise, ChatPairwise, CustomPairwise, DomainExpert, CustomPrompt