Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/allenai/vla-evaluation-harnessnpx agentmods add skills/allenai/vla-evaluation-harness/run-evaluationWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation)<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/run-evaluation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/run-evaluation"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/run-evaluation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 2 findings, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Privilege Escalation · line 69 Potential security issue detected. Manual review is recommended.Fix: Review the flagged content for security risks. Ensure no credentials, secrets, or sensitive data are exposed.
- medium MCP Rug Pull · line 34 Docker image references without a specific tag (:latest is implicit) or digest (@sha256:...) can be silently replaced by a malicious image.Fix: Pin the image: image:tag or image@sha256:abc123
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00027 | $0.01174 |
| Opus 5.5 | $0.00011 | $0.00470 |
| Sonnet 5.5 | $0.00005 | $0.00235 |
| Haiku 4.5 | $0.00003 | $0.00117 |
Grade A, and why
run-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 28d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 117 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Run an evaluation
Work from the target harness checkout. Read AGENTS.md, configs/README.md, and
the selected model/benchmark notes. Discover configs with
rg --files configs/model_servers configs/benchmarks. Use the relevant sections;
a validated setup need not be re-integrated.
1. Evaluation plan
Specify checkpoint/revision, benchmark version, tasks/variations, episodes, seeds, and initialization. Match observations, controller/actions, horizon, and replanning to the intended comparison; record deliberate differences.
Define expected work, recording needs, and a distinct output directory. Separate
smoke-run completion from reproduction evidence. docs/reproductions/running-guide.md
contains examples, not universal hardware allocations or verdict thresholds.
2. Preflight
- Verify checkpoint/auxiliary files, assets, image, and code actually executed.
Use
--devfor local benchmark edits or rebuild; keep model dependencies isolated. - Resolve asset roots through
src/vla_eval/dirs.pyand config mounts. Check paths on the consuming host; server and benchmark may not share storage. Reuse caches, coordinate acquisition, and provide writable scratch/output space and disk capacity. - Use allocated CPU/GPU resources and the permitted runtime. Consult
docs/runtimes.mdordocs/render-backends.mdwhen changing either; follow local policies. - Check server readiness via HTTP 200 from
/healthand connectivity from the benchmark runtime. Loading and first-inference compilation are separate phases; allow justified warm-up without masking a stall.
3. Baseline run
For an unfamiliar setup, run a small representative workload in separate terminals or managed processes:
# Server process; keep running during evaluation.
uv run vla-eval serve -c <model-config>
# Benchmark process, after the health check succeeds.
uv run vla-eval run -c <benchmark-config> \
--server-url ws://<server-host>:<port> --output-dir <run-directory>
Set actual task/episode counts through supported config/CLI options. Inspect images, state, actions, reset, termination, and recordings across relevant task paths and consecutive episodes.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 28d ago Changed · -62 lines · -60 tokens per session e8306055546c
- 1mo ago Changed e745e44660fb
- 2mo ago First seen · 179 lines · 87 tokens per session scan A d38bf678edda
run-evaluation is a skill published in the GitHub repository allenai/vla-evaluation-harness (661 stars, last pushed yesterday), licensed Apache-2.0. It adds 27 tokens to every session and 1,174 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
train-pose
Train/evaluate WiFi pose models honestly — camera-supervised (MediaPipe + CSI) and camera-free (WiFlow), always checked against the mean-pose baseline before any PCK is quoted.
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
experiment-audit
A review step that checks whether an experiment's reported results are supported by real data and a sufficiently broad test.
i4h-catheter-navigation-e2e
End-to-end smoke for catheter navigation covering setup, digital twin, DRR, and unit tests. Use when asked to run the full catheter workflow smoke or demo the v0.7 pipeline.
i4h-catheter-navigation-render-drr
Render a single DRR fluoroscopy frame from a CT cache or synthetic phantom. Use when asked to render DRR, generate a fluoro image, or smoke-test the Slang renderer.
i4h-catheter-navigation-smoke
Run CPU-only fluorosim smoke tests (imports, preprocessing, CLI parsers). Use when asked to smoke-test catheter navigation in CI or without a GPU.