A tool for scoring visual or multimodal task rollouts with a vision-language model and using the score as a pipeline gate. A rollout is one recorded attempt at completing a task.

It helps score individual rollouts, evaluate groups, run benchmark comparisons, choose thresholds, and gate generated outputs.

It turns subjective-looking rollout results into repeatable scores and prevents low-scoring results from moving through a pipeline.

vlm-eval — Skill for Claude Code and Codex by nebius

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add nebius/nebius-physical-ai --skill vlm-eval

Clone the repo

git clone --depth 1 https://github.com/nebius/nebius-physical-ai

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/nebius/nebius-physical-ai/vlm-eval/github.svg)](https://agentmods.dev/skills/nebius/nebius-physical-ai/vlm-eval)

Your own site

<a href="https://agentmods.dev/skills/nebius/nebius-physical-ai/vlm-eval"><img src="https://agentmods.dev/badge/skills/nebius/nebius-physical-ai/vlm-eval/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/nebius/nebius-physical-ai/vlm-eval"><img src="https://agentmods.dev/badge/skills/nebius/nebius-physical-ai/vlm-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 59 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,594 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits

NVIDIA SkillSpector pass 7 Sept 2026

How audits are shown

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00059	$0.01594
Opus 5	$0.00030	$0.00797
Sonnet 5	$0.00012	$0.00319
Haiku 4.5	$0.00006	$0.00159

Measured 4d ago against content hash a461507075ce, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-11, from the pricing page.

Security

Grade A, and why

vlm-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/tools/vlm-eval/SKILL.md · 132 lines

How it starts

The opening of the file, as written. The whole thing — 132 lines — stays where its author put it; the contents beside it link to each section on GitHub.

VLM eval (scoring rollouts and gating pipelines)

vlm-eval answers "did this rollout complete the task?" as a number, then turns that number into a gate. It is the judging half of the loop whose generating half is Cosmos/Genesis/Isaac rollouts and whose reasoning half is skills/tools/token-factory/SKILL.md.

Pick the right command

npa workbench vlm-eval run   --input-path <one-rollout>  --output-path <eval.json>
npa workbench vlm-eval loop  --input-path <prefix>       --output-path <prefix>
npa workbench vlm-eval benchmark --dataset <manifest> --output <report.json>
npa workbench vlm-eval status
npa workbench vlm-eval list
npa workbench vlm-eval workflow

run scores exactly one rollout. It discovers frames recursively, so if you point it at a prefix holding many rollouts they blend into a single meaningless score. That is the mistake loop exists to prevent: loop treats each directory under the prefix as its own rollout, scores each, and writes per-rollout results plus an aggregate task-success report.

benchmark sweeps configuration, not data. Use it to choose a threshold, rubric, or model against a labeled set before you trust any of them in a gate.

Backends

--backend is self-hosted (default), api, or stub.

self-hosted — an OpenAI-compatible server you run, addressed with --endpoint-url. This is the GPU-bearing path.
api — a hosted OpenAI-compatible endpoint; the key comes from the environment variable named by --api-key-env (default VLM_EVAL_API_KEY). Point this at Token Factory for a zero-GPU judge.
stub — deterministic, no model call. For wiring tests and CI only; a stub score is never evidence about a policy.

--endpoint-url accepts either a base URL or a full /chat/completions URL. Default model is Qwen/Qwen2-VL-7B-Instruct; --timeout-s defaults to 120.

Scoring controls that actually change the verdict

npa workbench vlm-eval run \
  --input-path s3://<bucket>/runs/<id>/rollout/ \
  --output-path s3://<bucket>/runs/<id>/eval.json \
  --task "pick and place the cube" \
  --backend api --model <model> --api-key-env NEBIUS_TOKEN_FACTORY_KEY \
  --frame-selection keyframes --max-frames 4 \
  --rubric-path ./rubric.txt \
  --success-threshold 0.8

Read the full file on GitHub · 132 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

4d ago Changed a461507075ce
8d ago First seen · 132 lines · 59 tokens per session scan A 3553631469d8

Subscribe to this mod's changes

vlm-eval is a skill published in the GitHub repository nebius/nebius-physical-ai (28 stars, last pushed yesterday), licensed Apache-2.0. It adds 59 tokens to every session and 1,594 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.

Analytics and metrics Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

vlm-eval