evalscope AGENTS.md

AGENTS.md

EvalScope — LLM evaluation framework with a registry-based plugin architecture. This file is the contract for AI coding agents working in this repo.

Setup

pip install -e .       # basic install
make dev               # dev + perf + docs extras + pre-commit

Python ≥ 3.10 (3.10 / 3.11 / 3.12). Dependencies: requirements/framework.txt + pyproject.toml [project.optional-dependencies] (extras: opencompass, vlmeval, rag, perf, app, aigc, sandbox, service, dev, docs, all, plus per-benchmark extras).

Build, lint, test

make lint                                                                       # apply Ruff fixes/formatting and run all pre-commit checks
pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings        # CI smoke test
pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s    # perf

Commits failing make lint are rejected on main.

Docs generation

Benchmark detail pages (docs/{zh,en}/benchmarks/<name>.md) and meta cache (evalscope/benchmarks/_meta/<name>.json) are auto-generated from each adapter's BenchmarkMeta.description + dataset statistics. Do not hand-edit those files.

Every BenchmarkMeta.description must be English Markdown with these sections in this order:

1. ## Overview: benchmark purpose and scope.
2. ## Task Description: bullet fields for Task Type, Input, Output, and Domain (use a more precise fourth field such as Modalities or Grading only when Domain does not apply).
3. ## Key Features: dataset scale/source, evaluated capabilities, and version-specific behavior.
4. ## Evaluation Notes: metrics, scoring procedure, runtime/dependency requirements, and compatibility limits.

Do not replace these required headings with benchmark-specific headings. Add extra sections only when the four required sections are insufficient.

When you add a benchmark or change its BenchmarkMeta.description, run: