Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add allenai/vla-evaluation-harness --skill add-benchmarkgit clone --depth 1 https://github.com/allenai/vla-evaluation-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/allenai/vla-evaluation-harness/add-benchmark)<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/add-benchmark"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/add-benchmark/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/allenai/vla-evaluation-harness/add-benchmark"><img src="https://agentmods.dev/badge/skills/allenai/vla-evaluation-harness/add-benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00029 | $0.01078 |
| Opus 5.5 | $0.00012 | $0.00431 |
| Sonnet 5.5 | $0.00006 | $0.00216 |
| Haiku 4.5 | $0.00003 | $0.00108 |
Grade A, and why
add-benchmark scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 28d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 96 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Add a benchmark
Work from the target harness checkout; paths below are relative to it. Read
AGENTS.md, CONTRIBUTING.md, and a comparable adapter/config. For an existing
integration, use the affected sections and preserve unrelated behavior.
1. Evaluation protocol
Trace the upstream evaluator: task selection → initialization → rollout → success check → cleanup. Establish:
- Version, task/variation set, initial-state selection, and seeds.
- Robot/controller, observations, action format, and control frequency.
- Warm-up, task-specific horizons, termination, success, and aggregation.
Inspect helper side effects: an expert rollout may select solvable states and initialize fields needed by success checks. Preserve those functions without blindly repeating expensive work. Bound setup retries; report failures instead of silently changing the evaluated population. Document intentional protocol deviations.
2. Runtime and assets
| Material | Placement and setup |
|---|---|
| Simulator dependencies | docker/Dockerfile.<name>; register in docker/build.sh and docker/push.sh. Pin compatible revisions and verify actual imports. |
| External scenes/data | Persistent cache via existing download mechanisms; record source, version, required contents, and acquisition command. |
| Generated files | Writable scratch/output paths, separate from immutable assets; libraries may try to write beside installed assets. |
Use vla_eval.dirs.assets_cache("<benchmark>"). Its root follows
VLA_EVAL_ASSETS_CACHE → VLA_EVAL_HOME/assets →
${XDG_CACHE_HOME:-~/.cache}/vla-eval/assets. Config mounts must map the host cache
to the simulator's container path with the same precedence.
For a working mount/download example, inspect
configs/benchmarks/behavior1k/eval.yaml and Behavior1KBenchmark._ensure_assets
in src/vla_eval/benchmarks/behavior1k/benchmark.py.
- Reuse complete caches; coordinate lazy downloads or prepare them before shards.
- Use
ensure_licensefor gated assets. User acceptance is passed through--accept-license <id>orVLA_EVAL_ACCEPTED_LICENSES; do not assume consent. - Check the intended runtime's write permissions, including read-only images.
- Follow
docs/render-backends.md; verify real camera output before declaring renderer support. Successful environment creation is insufficient.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 28d ago Changed · -134 lines · -49 tokens per session dc5f76dd029f
- 2mo ago First seen · 230 lines · 78 tokens per session scan A 7be89501e6b0
add-benchmark is a skill published in the GitHub repository allenai/vla-evaluation-harness (661 stars, last pushed yesterday), licensed Apache-2.0. It adds 29 tokens to every session and 1,078 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
train-pose
Train/evaluate WiFi pose models honestly — camera-supervised (MediaPipe + CSI) and camera-free (WiFlow), always checked against the mean-pose baseline before any PCK is quoted.
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
experiment-audit
A review step that checks whether an experiment's reported results are supported by real data and a sufficiently broad test.
i4h-catheter-navigation-e2e
End-to-end smoke for catheter navigation covering setup, digital twin, DRR, and unit tests. Use when asked to run the full catheter workflow smoke or demo the v0.7 pipeline.
i4h-catheter-navigation-render-drr
Render a single DRR fluoroscopy frame from a CT cache or synthetic phantom. Use when asked to render DRR, generate a fluoro image, or smoke-test the Slang renderer.
i4h-catheter-navigation-smoke
Run CPU-only fluorosim smoke tests (imports, preprocessing, CLI parsers). Use when asked to smoke-test catheter navigation in CI or without a GPU.