harness-evals AGENTS.md

AGENTS.md - harness-evals

Project Overview

harness-evals is an open-source AI evaluation framework for LLM agents, prompts, and structured outputs. It provides a pip install-able scoring engine with 70+ metrics across deterministic, structural, operational, reliability, predictability, MCP, similarity, LLM-judged, RAG, safety, agent, conversation, and security categories.

Core principle: An eval always produces a Score. Every metric is a single class with a measure() method.

Data flow: Golden (authored) + agent output -> EvalCase (evaluated) -> Score (result)

Language: Python 3.10+ License: Apache 2.0 Package name: harness-evals Import name: harness_evals

Build System

pip install -e "."                # Core only (deterministic metrics, no LLM key needed)
pip install -e ".[llm]"           # + OpenAI, Anthropic for LLM-judged metrics
pip install -e ".[otlp]"          # + OTLP metrics & traces export
pip install -e ".[langfuse]"      # + Langfuse source/sink
pip install -e ".[similarity]"    # + BLEU metric (nltk)
pip install -e ".[harness]"       # + Harness AI Service LLM provider
pip install -e ".[benchmarks]"    # + Academic benchmarks (MMLU, GSM8K, HumanEval, etc.)
pip install -e ".[all]"           # Everything
pip install -e ".[all,dev]"       # Everything + dev tools

Build tool: Poetry via pyproject.toml (backend: poetry-core) No compiled extensions — pure Python.

Testing

pytest tests/ -v                              # All tests
pytest tests/ -v -m unit                      # Unit tests only
pytest tests/metrics/ -v                      # Specific directory
pytest tests/test_core.py -v                  # Specific file
pytest tests/test_core.py::test_evaluate -v   # Specific function
pytest tests/ --cov=harness_evals --cov-report=html  # With coverage

• Mark tests: @pytest.mark.unit, @pytest.mark.integration
• Test data: tests/data/

ALWAYS run pytest tests/ -v before committing.