vla-evaluation-harness: Instructions file for Codex

AGENTS.md

This file provides context for AI coding assistants working on this repository.

Project Overview

vla-evaluation-harness (vla-eval) is a unified evaluation framework for Vision-Language-Action (VLA) models across 11+ robot simulation benchmarks. Models integrate once, benchmarks integrate once, and the full cross-evaluation matrix works automatically.

Core design: Model server communicates with benchmark (Docker container, with optional GPU access for rendering) via WebSocket + msgpack binary protocol. This decouples model dependencies from benchmark dependencies entirely.

Commands

# Setup
uv sync --python 3.11 --all-extras --dev

# Quality (CI runs these on every PR)
make lint          # ruff check --fix + ruff format
make check         # ruff check + ruff format --check + ty check (no auto-fix)
make test          # uv run pytest

# Single test
uv run pytest tests/test_protocol.py -v
uv run pytest tests/test_protocol.py::test_name -v

# Smoke tests (model servers, benchmarks, config validation)
vla-eval test                                       # validate configs only (fast, default)
vla-eval test --all                                 # run all categories (validate + server + benchmark)
vla-eval test --list                                # show available tests + prerequisites
vla-eval test --server                              # smoke-test all model servers
vla-eval test --benchmark                           # smoke-test all benchmarks
vla-eval test -c configs/model_servers/cogact.yaml  # smoke-test a specific config
make smoke                                          # shortcut for vla-eval test --all

Line length is 119 (configured in pyproject.toml for ruff and ty).

Architecture

CLI (cli/main.py)
 └─ Orchestrator (orchestrator.py)
     ├─ Benchmark (benchmarks/base.py)  ── runs inside Docker container
     │   └─ EpisodeRunner (runners/)    ── sync or live
     │       └─ Connection (connection.py) ←─ WebSocket/msgpack ─→ ModelServer (model_servers/base.py)
     ├─ ResultCollector (results/collector.py)  ── in-memory aggregation for stdout summary
     └─ RecordingStore (recording.py) ── SQLite (one per eval) capturing step rows + per-episode results