auto-codabench CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

autocodabench: a pip-installable library for agentic authoring + pre-launch validation of Codabench competition bundles, built on the Claude Agent SDK. Target venue: JMLR MLOSS (see docs/ and the design discussion on branch jmlr-oss-direction). The package lives in src/autocodabench/; a Chainlit web UI (web/, deployed as an HF Space via Dockerfile) consumes the library; benchmark/ holds the pure-SDK end-to-end benchmarks (create-bench and validate-bench, both live under benchmark/).

The root README.md doubles as the HF Spaces metadata file — its YAML header configures the Space; don't remove it.

Commands

pip install -e .                          # editable install (also: pip install -e '.[dev]')
python -m pytest tests/                   # unit suite — fast, fully keyless, must stay that way

# Keyless CLI paths (work with no Claude auth at all):
autocodabench demo --out /tmp/demo        # rebuild+validate the demo bundle from a recorded run
autocodabench validate <bundle-dir-or-zip> [--facts facts.yaml]
autocodabench checks list                 # registered checks by tier, with citations

# Auth-requiring paths (subscription login preferred; ANTHROPIC_API_KEY second):
autocodabench auth status [--no-probe]    # active path + masked creds; verifies the SDK can sign in (live turn) unless --no-probe
autocodabench auth use <auto|subscription|api_key>   # choose; subscription hides any key from the SDK
autocodabench validate <bundle> --judged      # adds LLM-judged advisory checks
autocodabench plan-build-validate "<idea>" [--data D] [--pdf P]   # agentic plan→build→validate pipeline (alias: create; idea and/or a PDF proposal)
autocodabench plan "<idea>" [--data D]   # Phase 1 only → specs/implementation_plan.md
autocodabench build <plan.md | --run-dir D>  # Phase 2 only → build a bundle from a plan

# Any agentic command above accepts --backend: claude[:model] (default), ollama:<m>, openai:<m>, or URL#<m>.

python -m autocodabench.core.bundle_io    # core smoke test (demo bundle in a tempdir)
python -m autocodabench.mcp.server        # MCP stdio server (hangs on stdin — correct)
python scripts/make_demo_fixture.py       # regenerate the shipped replay fixture

cd web && chainlit run app.py --host 127.0.0.1 --port 8500 -h   # web UI (needs .env)

# Deploy the web UI to the Hugging Face Space (GitHub master is the source of
# truth; the script injects the HF README header and force-pushes to hf main):
scripts/deploy_hf.sh [--dry-run] [--yes] [src-ref]   # default src-ref: origin/master

# Benchmarks (pure-SDK; any backbone via --backend; needs Docker + a populated instrument):
python benchmark/autocodabench_create_bench/run.py --competition style-trans-fair --backend claude