inspect-robots CLAUDE.md

Inspect Robots — agent guide

Inspect Robots is the "Inspect AI for robotics": an open-source evaluation framework for physical AI / VLA (vision-language-action) models. This repo is the framework; concrete benchmarks and backend adapters live elsewhere (see below).

The one big idea

LLM evals have one swappable input (the model). Robotics evals have two:

• Policy (the VLA "brain") — observation → ActionChunk (open-loop horizon).
• Embodiment (the robot/sim "body + world") — executes actions, owns spaces.

A Task (a dataset of Scenes + scorers) is defined independently of both. eval() checks a (policy, embodiment) pair is compatible, runs the closed-loop rollout, scores it, and writes an immutable EvalLog. Mirrors Inspect AI's Task = dataset + solver + scorer, eval(), EvalLog, registry/decorator model.

Layout

• src/inspect_robots/ — the package (see src/inspect_robots/CLAUDE.md for the module map).
• tests/ — pytest; the CubePick mock world exercises the whole stack with no hardware or sim.
• plans/ — design docs. plans/0001-foundation-design.md is the authoritative spec (read its §9–§11 "binding resolutions" before changing core interfaces).
• examples/ — runnable demos (quickstart.py).
• plugins/* — first-party plugin packages (concrete sims/VLAs that are out of scope for the numpy-only core), each its own package with its own pyproject, entry point, tests, and coverage scope. A uv workspace ([tool.uv.workspace]) ties them in: uv sync --all-packages --extra dev installs core + all plugins editable. They never count toward the core 100% gate (coverage is scoped to inspect_robots). E.g. plugins/inspect-robots-isaacsim/ (Isaac Lab embodiment), plugins/inspect-robots-ros/ (ROS 1/ROS 2 embodiment speaking rosbridge with no ROS dependency), plugins/inspect-robots-xpolicylab/ (policy adapter speaking the XPolicyLab websocket protocol — 40+ served VLAs, no xpolicylab dep), and plugins/inspect-robots-agent/ (LLMs as policies via the OpenAI-compatible wire formats or Anthropic's native Messages API — httpx only, no provider SDKs; registered as agent), and plugins/inspect-robots-capx/ (CaP-X code-as-policy over perception and IK HTTP servers; registered as capx), and plugins/inspect-robots-voice/ (local microphone transcription with Parakeet and local Kokoro policy narration; registered as the voice operator input and speaker sink).