inspect_evals: Instructions file for Codex

AGENTS.md

This is the inspect_evals repository - a collection of evaluation tasks for the Inspect AI framework.

These guidelines are for AI coding agents using evals or preparing contributions. Human contributors please see CONTRIBUTING.md.

Setup Commands

This project uses uv (by Astral) for package management. Do not use pip install, python -m venv, source .venv/bin/activate, or bare python/pytest commands. Always use uv to run commands:

• uv sync — install/sync dependencies (replaces pip install -e .)
• uv run pytest ... — run tests (not pytest or python -m pytest)
• uv run python ... — run Python scripts (not python or python3)
• uv run inspect eval ... — run evaluations
• uv run ruff ... — run the linter
• uv run mypy ... — run the type checker

Useful Commands

To run evaluations, run uv run inspect eval inspect_evals/eval_name.

• To run a specific task (i.e, a function with the @task decorator), run uv run inspect eval inspect_evals/eval_name@task_name.
• Some useful arguments are:
  1. --limit X: Run only up to X samples.
  2. --model model_name: Run on this particular model. Multiple models can be run with commas like model1,model2. You can find information on model names here. If the user lacks an API key for OpenAI, Anthropic, or Google, note this down in your report. You should assume the user has these keys until Inspect throws an error.
  3. -T allows you to pass task arguments in from the command line.

Testing instructions

To validate changes develop pytests that tests for erroneous behaviour and run the tests after your fix.

Additionally for quality control tests are mandatory to run:

1. Quick checks for general adherance. Run
   • uv run ruff check, uv run ruff format --check, uv run mypy src tests, and uv run inspect-evals-lint eval_name for the eval you changed. (These come from .github/workflows/checks.yml)
   • Run the inspect-evals-lint.
2. Running unit tests specific to an eval
   • Install dev and test dependency groups which are not installed by defaultuv sync --group dev --group test
   • uv run pytest tests/eval_name. For an eval under packages/, whose dependencies conflict with the root environment, run uv run --group dev tox -e eval_name instead.
   • Whenever possible, run tests marked slow, dataset_download, k8s, gpu, arxiv_smoke, llm_smoke, or openrouter_smoke which are skipped by default unless you enable them.
   • Write custom tests to verify your fix and add the subset that are useful for enforcing intended behaviour. Mark any test you add according to Apply Pytest Marks. Tests must be deterministic and must not reach the network: use mockllm/model for model output, unittest.mock for external APIs, and tmp_path for files.
3. After tests pass, run the evaluation on a few sample.
   • Read the transcript as well as the score.
4. Run make check before opening a PR to run linters and file auto-generation.