Borrowing it
Nothing to install: this file belongs to allenai/vla-evaluation-harness. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/allenai/vla-evaluation-harness/main/.claude/skills/run-evaluation/SKILL.mdgit clone --depth 1 https://github.com/allenai/vla-evaluation-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation)<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/run-evaluation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/run-evaluation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 2 findings, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Privilege Escalation · line 69 Potential security issue detected. Manual review is recommended.Fix: Review the flagged content for security risks. Ensure no credentials, secrets, or sensitive data are exposed.
- medium MCP Rug Pull · line 34 Docker image references without a specific tag (:latest is implicit) or digest (@sha256:...) can be silently replaced by a malicious image.Fix: Pin the image: image:tag or image@sha256:abc123
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00087 | $0.02006 |
| Opus 5 | $0.00044 | $0.01003 |
| Sonnet 5 | $0.00017 | $0.00401 |
| Haiku 4.5 | $0.00009 | $0.00201 |
Grade A, and why
run-evaluation scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
Wait until `curl -fsS http://localhost:8000/health` returns HTTP 200 — the server only starts listening after `__init__` finishes loading weights, so this is the readiness signal. How it starts
The opening of the file, as written. The whole thing — 179 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Run Evaluation
Evaluate a VLA model against a simulation benchmark. The harness decouples model serving (WebSocket server) from benchmark execution (Docker container), so they run as two separate processes.
1. Identify the config pair
Every evaluation needs two YAML configs:
- Model server config (
configs/model_servers/<model>.yaml) — definesscriptandargsfor the model server - Benchmark config (
configs/<benchmark>.yaml) — definesdocker.image,benchmarksentries, andoutput_dir
List available configs:
ls configs/model_servers/ # model servers
ls configs/*.yaml # benchmarks
Not all model–benchmark pairs are compatible. The model server must produce actions in the format the benchmark expects (e.g. 7-DoF for LIBERO). Many model configs encode their target benchmark in the filename (e.g. oft_libero.yaml, xvla_calvin.yaml).
2. Check prerequisites
| Requirement | Check command | Notes |
|---|---|---|
| uv | which uv |
Runs model server in isolated env |
| Docker | docker info |
Benchmarks run inside containers |
| GPU | nvidia-smi |
Model inference + sim rendering |
| Disk space | df -h |
Model weights (tens of GB) + Docker images (4–10 GB each) |
Model weights download automatically on first vla-eval serve. Docker images are pulled on first vla-eval run (or pre-pull with docker pull <image>).
Docker image rebuild: Benchmark code runs inside the Docker image. If you (or someone else) changed benchmark source code in src/vla_eval/benchmarks/, the pre-built image is stale — you must rebuild before running:
./docker/build.sh <benchmark_name> # e.g. ./docker/build.sh libero
Skip the rebuild only if using --dev mode, which bind-mounts local src/ into the container.
3. Run the evaluation (two terminals)
The model server and benchmark runner communicate over WebSocket and must run concurrently.
Terminal 1 — start the model server:
vla-eval serve -c configs/model_servers/<model>.yaml
Wait until curl -fsS http://localhost:8000/health returns HTTP 200 — the server only starts listening after __init__ finishes loading weights, so this is the readiness signal.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 179 lines · 87 tokens per session scan A d38bf678edda
run-evaluation is a skill published in the GitHub repository allenai/vla-evaluation-harness (591 stars, last pushed 7d ago), licensed Apache-2.0. It adds 87 tokens to every session and 2,006 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
train-pose
Train/evaluate WiFi pose models honestly — camera-supervised (MediaPipe + CSI) and camera-free (WiFlow), always checked against the mean-pose baseline before any PCK is quoted.
experiment-audit
A review step that checks whether an experiment's reported results are supported by real data and a sufficiently broad test.
i4h-catheter-navigation-e2e
End-to-end smoke for catheter navigation covering setup, digital twin, DRR, and unit tests. Use when asked to run the full catheter workflow smoke or demo the v0.7 pipeline.
i4h-catheter-navigation-render-drr
Render a single DRR fluoroscopy frame from a CT cache or synthetic phantom. Use when asked to render DRR, generate a fluoro image, or smoke-test the Slang renderer.
i4h-catheter-navigation-smoke
Run CPU-only fluorosim smoke tests (imports, preprocessing, CLI parsers). Use when asked to smoke-test catheter navigation in CI or without a GPU.