Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/vasuag09/harness-claudenpx agentmods add skills/vasuag09/harness-claude/benchmarkWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/vasuag09/harness-claude/benchmark)<a href="https://agentmods.dev/skills/vasuag09/harness-claude/benchmark"><img src="https://agentmods.dev/badge/skills/vasuag09/harness-claude/benchmark.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00088 | $0.01136 |
| Opus 5 | $0.00044 | $0.00568 |
| Sonnet 5 | $0.00018 | $0.00227 |
| Haiku 4.5 | $0.00009 | $0.00114 |
Grade A, and why
benchmark scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 75 lines — stays where its author put it; the contents beside it link to each section on GitHub.
/benchmark — does this component earn its keep?
Goal: replace opinion with evidence. Run a task k times with a named component enabled and
k times without it, each trial in its own throwaway git worktree, then compare reliability.
Report pass@k (passed in ≥1 trial — "can it work") and pass^k (passed in all k trials —
"does it work reliably"), and a verdict.
Opt-in. No default-pipeline skill or hook invokes
/benchmark. It forks git worktrees and (when you supply a real task) spawns work k×2 times — always an explicit, deliberate run. Git boundary: it onlygit worktree add/remove(an isolated sandbox); it never commits, pushes, or branches the working tree.
When to run
- You want to decide whether a skill / agent / rule is worth keeping (keep-vs-cut on data).
- A
/extractproposal sits at R5 MANUAL and you want a real empirical signal — benchmark it with--component <draft-name>so the result lands where R5 looks.
The harness contract
scripts/eval/benchmark.js is generic: it knows nothing about your component. It toggles the
component only through env vars the task command honors, for each trial:
| env | meaning |
|---|---|
HARNESS_COMPONENT |
the component name (label) |
HARNESS_COMPONENT_ENABLED |
1 (with) · 0 (without) |
HARNESS_TRIAL |
0-based trial index |
A trial passes iff its score command exits 0. Default score = the task's own exit code;
recommended = node scripts/eval/checkpoint.js <spec> so a trial is scored against a spec's
acceptance criteria (reusing the v0.4 gate).
Do this
- Define the task. Decide what "doing the task" means and write a command that performs it
in the current worktree, reading
HARNESS_COMPONENT_ENABLEDto enable/disable the component under test (e.g. include the skill's guidance only when=1). For a model-driven task, the command drives a subagent; for a deterministic check, any shell script works. - Define scoring. Either let the task's exit code be the signal, or pass
--score(e.g.checkpoint.jsagainst the task's spec) so PASS/FAIL is judged against acceptance criteria. - Run the benchmark:
Exit 0 = pass verdict (component earns its keep) · 1 = fail verdict (no reliability gain) · 2 = usage/setup error (nonode scripts/eval/benchmark.js --component <name> --task '<cmd>' [--score '<cmd>'] [--k 5]--task, not a git repo, no HEAD commit). - Read the result. The run prints both cohorts' pass@k / pass^k / success-rate and the
verdict, and writes
.claude/eval/benchmarks/<slug>.json(<slug>= the component name). The verdict is: earns its keep iffwith.successRate > without.successRateor (with.passCaret && !without.passCaret). - Act on it. Recommend keep or cut with the numbers. For an
/extractproposal, the artifact now feeds R5: re-runnode scripts/eval/extract-rubric.js <draft>and R5 reports PASS/FAIL from the benchmark instead of MANUAL. Promotion intoskills/is still a human call (R4 + your approval) — a passing benchmark never auto-promotes.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 7d ago First seen · 75 lines · 88 tokens per session scan A 3a2bbcc16caf
benchmark is a skill published in the GitHub repository vasuag09/harness-claude (2 stars, last pushed 2mo ago), licensed MIT. It adds 88 tokens to every session and 1,136 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ci-tests
Run the test suite for the current repo, auto-detecting Python (pytest/uv), Node (vitest/pnpm), or Rust (cargo test).
git-worktree
Create isolated git worktrees for feature development without switching branches.
git-worktree-status
Check status of background verification tasks running in a git worktree.
typescript-providers
Implement, modify, test, or document TypeScript provider packages under ts/packages/providers, including framework adapters for OpenAI, Anthropic, Google, LangChain, Mastra, Vercel, LlamaIndex, Cloudflare, and Claude Agent SDK. Use for provider-specific TS work; do not use for core-only changes.
typescript-sdk
Implement or modify TypeScript SDK behavior in @composio/core or shared TypeScript packages, including tools, toolkits, sessions, auth configs, connected accounts, modifiers, and generated SDK surfaces. Use for TS runtime/API work; pair with typescript-testing for verification and cross-sdk-parity when Python must…
typescript-testing
Select and run TypeScript SDK verification for packages, examples, type checks, linting, builds, Vitest suites, and runtime E2E tests. Use when adding tests, diagnosing TypeScript CI, choosing a focused test command, or validating TypeScript package changes. Do not use for Python-only checks.