Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Ahmad-Jaradat-Space/co-scientist-plugin --skill benchmarking-modelsgit clone --depth 1 https://github.com/Ahmad-Jaradat-Space/co-scientist-pluginWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ahmad-jaradat-space/co-scientist-plugin/benchmarking-models)<a href="https://agentmods.dev/skills/ahmad-jaradat-space/co-scientist-plugin/benchmarking-models"><img src="https://agentmods.dev/badge/skills/ahmad-jaradat-space/co-scientist-plugin/benchmarking-models/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ahmad-jaradat-space/co-scientist-plugin/benchmarking-models"><img src="https://agentmods.dev/badge/skills/ahmad-jaradat-space/co-scientist-plugin/benchmarking-models.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00097 | $0.01009 |
| Opus 5 | $0.00048 | $0.00504 |
| Sonnet 5 | $0.00019 | $0.00202 |
| Haiku 4.5 | $0.00010 | $0.00101 |
Grade A, and why
benchmarking-models scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 95 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Benchmarking models
The bench runs N candidate models on one research goal and settles them in a cross-model Elo tournament judged by a separate model. It is the tool for "which model should I use for this", and for checking that a change to prompts or config actually helped.
This runs through the CLI, not the MCP tools. It runs a full pipeline per candidate, so it is several times the size of a single session.
Before running
- Check the backend with
co-scientist:doctorand readbilling. - Size the run. It scales as candidates times
--ntimes--matches, so five candidates is five full generation passes plus every pairing between them. - Read the caps correctly.
--budget-per-candidateand--judge-budgetmultiply: five candidates at 3.0 plus a judge at 5.0 is a cap of 20.0. Undersubscriptionthose are usage caps in the engine's odometer units, not prices, and nothing is charged; undermeteredthey are real dollars. - Get an explicit go-ahead on the size.
Running
Reproduce the paper's comparison:
co-scientist bench "Identify hypotheses about X" \
--preset paper \
--judge claude_cli:sonnet
Compare your own candidates:
co-scientist bench "Identify hypotheses about X" \
-c opus=claude_cli:opus \
-c sonnet=claude_cli:sonnet \
-c codex=codex_cli:gpt-5.6-sol \
--judge claude_cli:sonnet
Both candidates there run on a subscription, so nothing is billed. Add
--budget-per-candidate and --judge-budget (USD caps, defaults 3 and 5) only
when a candidate or the judge is a metered API model.
Candidate syntax is label=backend:model[@mode]. Mode pipeline is the
default and runs the full agent loop. Mode direct is a single raw model call
with no tools, which is the honest baseline for "does the multi-agent structure
earn the extra calls".
Useful flags:
--nhypotheses per candidate, default 2--matchestournament matches per pair, default 2--goldsetscores against a known answer key;--preset paper-amlbundles one. Passnoneto disable a preset's default.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 95 lines · 97 tokens per session scan A 9ef466ba563b
benchmarking-models is a skill published in the GitHub repository Ahmad-Jaradat-Space/co-scientist-plugin (0 stars, last pushed 21d ago), licensed Apache-2.0. It adds 97 tokens to every session and 1,009 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
dataset-transformation
Generates code that transforms datasets between ML schemas for model training or evaluation. Use when the user says "transform", "convert", "reformat", "change the format", or when a dataset's schema needs to change to match the target format — always use this skill for format changes rather than writing inline…
dataset-evaluation
Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR). Use when the user says "is my dataset okay", "evaluate my data", "check my training data", "I have my own data", or before starting any fine-tuning job. Detects file format, checks schema compliance against the selected model…
use-case-specification
Creates a reusable use case specification file that defines the business problem, stakeholders, and measurable success criteria for model customization, as recommended by the AWS Responsible AI Lens. Use as the default first step in any model customization plan. Skip only if the user explicitly declines or already has…
model-selection
Selects a base model for the user's use case by querying SageMaker Hub. Use when the user asks which model to use, wants to select or change their base model, mentions a model name or family (e.g., "Llama", "Mistral", "Nova"), or wants to evaluate a base model — always activate even for known model names because the…
finetuning-technique
Selects a fine-tuning technique (SFT, DPO, RLVR, or RLAIF) for the user's use case and validates it against the selected model's available recipes. Use when the user has decided to finetune and needs to choose a technique, or when the technique needs to be validated against a model. Requires a base model to already be…
install-openviking-memory
Install and configure the OpenViking long-term memory plugin for OpenClaw via natural conversation. Once installed, the plugin automatically captures facts from chats and recalls relevant context before each reply (auto-capture + auto-recall, cross-session). Covers prerequisites, install through OpenClaw's plugin…