scorio AGENTS.md

AGENTS.md

This file provides guidance to coding agents working with code in this repository.

Overview

Scorio is a research toolkit for Bayesian evaluation and ranking of LLMs (especially reasoning models under test-time scaling / repeated sampling). It backs three papers (ICLR 2026, ACL 2026, an arXiv preprint on test-time scaling) and ships companion HuggingFace datasets of reasoning traces (notebooks/datasets/{trace,lite,math,gpqa}).

The repo contains three parallel implementations of the same API, kept in version lockstep via the root VERSION file and scripts/sync_version.py:

• scorio/ — Python, the reference implementation (published to PyPI)
• julia/Scorio.jl/ — Julia, mirrors the Python eval/rank/aggregate/sinf/utils API (registered in the Julia General Registry)
• js/scorio/ — TypeScript, zero-dependency port (npm), currently centered on the eval APIs

When changing the public API or fixing a bug in one implementation, check whether the equivalent should also change in the others.

Common commands

All commands below run from the repo root unless noted; see the root Makefile for the authoritative list.

Python (scorio/)

pip install -e ".[dev]"          # install with dev deps (pytest, ruff, mypy, sphinx...)

make test                        # pytest tests/
pytest tests/eval                # a single test dir
pytest tests/rank -m "not slow"  # rank tests excluding slow ones (make test-rank-py)
pytest tests/rank -m slow        # slow rank tests only (make test-rank-py-slow)
pytest tests/eval/test_bayes.py::test_name -v   # a single test

make format-check                # ruff format --check + ruff check
make format                      # ruff format + ruff check --fix
make lint                        # ruff check + mypy

make py-docs-build               # Sphinx docs -> docs/_build/html
make py-docs-serve                # serve built docs on :4000

Pytest marker slow is registered in pyproject.toml for heavy ranking APIs — use it on any new test that's expensive (MCMC, MILP solvers, etc.).