Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/bgauryy/octocodenpx agentmods add skills/bgauryy/octocode/octocode-benchmarkWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/bgauryy/octocode/octocode-benchmark)<a href="https://agentmods.dev/skills/bgauryy/octocode/octocode-benchmark"><img src="https://agentmods.dev/badge/skills/bgauryy/octocode/octocode-benchmark/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/bgauryy/octocode/octocode-benchmark"><img src="https://agentmods.dev/badge/skills/bgauryy/octocode/octocode-benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 2 findings, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium MCP Rug Pull · line 38 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
- medium MCP Rug Pull · line 48 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00117 | $0.02215 |
| Opus 5 | $0.00059 | $0.01107 |
| Sonnet 5 | $0.00023 | $0.00443 |
| Haiku 4.5 | $0.00012 | $0.00221 |
Grade A, and why
octocode-benchmark scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 134 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Octocode benchmark
Plain-markdown, run-by-hand CLI research comparison. Octocode is the anchor; each baseline
is a separate pairwise matchup (octocode vs rtk | headroom | gh). Per question,
per pass: two isolated runners answer, one blind judge grades them X / Y (randomized per
question). Run ≥3 passes; the rollup shows every matchup together. No harness, no JSON.
Paths below are relative to the package root packages/octocode-benchmark/. Shared tooling
lives in compare/bin/; questions in compare/github-questions/; reports in results/.
Flow (4 phases)
- Preflight — verify + pin every arm; a failure invalidates the run.
- Answer — 2 isolated runners (anchor + baseline) per question/pass, leanest-legal path, each appends a
## Q<n>section toanswers/<arm>-p<pass>.md. - Judge — after both sections exist, one blind judge reasons to a verdict, then scores.
- Summarize — validate logs, aggregate paired stats, update the rollup.
How it is MEASURED (characters, never self-reported)
The metric is total_chars = model-in + model-out in Unicode code points, from an
instrumented log — the tool transcript only (excludes system prompt, tool schemas, model
reasoning; the fixed per-arm primer is excluded by rule; any later help/schema/failed call
is counted).
- model-out = the command string + args the model wrote, plus the final answer.
- model-in = the tool output pulled back into context (for Headroom, the compressed output).
Mechanism: every research command runs through its arm's thin wrapper, which shells the real CLI unchanged, prints output verbatim, and appends one JSONL row per call:
| Arm | Wrapper | Runs | Log env |
|---|---|---|---|
| octocode (local build) | compare/bin/octoc |
npx octocode tools … |
OCTO_LOG |
| octocode (published pin) | compare/bin/octoc1822 |
npx -y [email protected] tools … |
OCTO_LOG |
| gh+RTK | compare/bin/rtkm |
rtk gh … |
RTK_LOG |
| gh+Headroom | compare/bin/ghc |
gh … → Headroom compress |
GHC_LOG (+ HR_PY) |
| plain gh | compare/bin/ghm |
gh … (read-only) |
— |
What ships with it
22 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- README.md 577 B
- references/aggregation-and-stats.md 5.0 KB
- references/BENCHMARK.md 4.9 KB
- references/example-verdict.md 2.7 KB
- references/INSTRUCTIONS.md 2.4 KB
- references/JUDGING.md 4.9 KB
- references/matchup-readme.md 1.9 KB
- references/primer-gh-headroom.md 2.7 KB
- references/primer-gh-rtk.md 2.4 KB
- references/primer-gh.md 2.1 KB
- references/primer-octocode.md 4.6 KB
- references/REPORT_TEMPLATE.md 2.5 KB
- references/run-phases.md 2.8 KB
- references/run-preflight.md 2.9 KB
- references/run-with-agents.md 790 B
- references/RUNNER_TOOL_CONTEXT.md 1.2 KB
- references/RUNNER.md 3.4 KB
- references/SCORING.md 3.7 KB
- scripts/_hr_check.py 803 B runs code
- scripts/_hr_import.py 30 B runs code
- scripts/check-prereqs.sh 4.7 KB runs code
- scripts/measure.sh 1.7 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 134 lines · 117 tokens per session scan A cef37ebcb5fd
octocode-benchmark is a skill published in the GitHub repository bgauryy/octocode (932 stars, last pushed yesterday), licensed MIT. It adds 117 tokens to every session and 2,215 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
component-creation
Step-by-step workflow for creating accessible, tested UI components. Use when the user asks to create a new UI component.
fix-tests
Systematic workflow for diagnosing and fixing failing tests. Use when the user reports failing tests or asks to fix test failures.
ci-cd
Workflow for creating a complete CI/CD pipeline. Use when the user needs to set up or modify CI/CD pipelines.
coverage-improvement
Systematic workflow for identifying coverage gaps and writing missing tests. Use when coverage is below target or the user asks to improve coverage.
debug-issue
Structured workflow for debugging a reported or reproduced issue. Use when the user asks to debug a bug, find root cause, or fix a failing scenario.
sparc-methodology
SPARC (Specification, Pseudocode, Architecture, Refinement, Completion) comprehensive development methodology with multi-agent orchestration.