Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add RBraga01/builder-ai --skill model-benchmarkinggit clone --depth 1 https://github.com/RBraga01/builder-aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/rbraga01/builder-ai/model-benchmarking)<a href="https://agentmods.dev/skills/rbraga01/builder-ai/model-benchmarking"><img src="https://agentmods.dev/badge/skills/rbraga01/builder-ai/model-benchmarking/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/rbraga01/builder-ai/model-benchmarking"><img src="https://agentmods.dev/badge/skills/rbraga01/builder-ai/model-benchmarking.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00043 | $0.01254 |
| Opus 5.5 | $0.00017 | $0.00502 |
| Sonnet 5.5 | $0.00009 | $0.00251 |
| Haiku 4.5 | $0.00004 | $0.00125 |
Grade A, and why
model-benchmarking scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 121 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Model Benchmarking
The Law
A MODEL IS NOT CHOSEN UNTIL IT HAS BEEN TESTED ON YOUR TASK DATA.
Leaderboard scores are averaged over tasks you're not building.
"It's the best model" is a claim about benchmarks someone else ran.
Task data + defined metric + three tested tiers IS a model selection.
When to Use
Trigger when:
- Choosing a model for any new production feature
- Considering a switch to a newer, cheaper, or faster model
- Validating that a smaller model can replace a larger one
- Comparing providers (OpenAI, Anthropic, Mistral, local)
When NOT to Use
- Initial feasibility exploration (before task definition is stable) — benchmark when you know what you're measuring
- Model compatibility checks (does this model support tool use, JSON mode, etc.) — that's a capability query, not a benchmark
- Leaderboard research to narrow the candidate list — that's input to Stage 3, not a substitute for it
The Process
Four stages. Do not collapse them.
Stage 1 — Define the Bar
Before running any model, write down:
| Decision | What to Define |
|---|---|
| Task | Exact input format, exact output format, edge cases |
| Metric | Accuracy, faithfulness score, extraction F1, LLM-as-judge, task pass rate |
| Pass threshold | The minimum score to go to production (e.g., ≥ 88%) |
| Budget constraint | Max cost per 1k calls, max monthly spend |
| Latency constraint | Max acceptable p95 (e.g., ≤ 2.0s) |
If you cannot define the metric and threshold first, the selection criteria are not clear enough to proceed.
Stage 2 — Build a Representative Test Set
- Minimum 50 examples; 200+ for features used at volume
- Label ground truth before running any model. Seeing outputs first contaminates labels.
- Include: easy cases, hard cases, edge cases, adversarial inputs
- Tag each example by difficulty:
easy / medium / hard - Store in
benchmarks/<feature>/test-set.jsonl
Stage 3 — Benchmark at Least Three Tiers
Always test at least one tier below your expected choice:
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 121 lines · 43 tokens per session scan A 7cd95fd00299
model-benchmarking is a skill published in the GitHub repository RBraga01/builder-ai (2 stars, last pushed 10d ago), licensed MIT. It adds 43 tokens to every session and 1,254 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ai-prompt-regression-testing
Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.
recipe-eval-prompt
Compares original and optimized prompts through repeated blind paired execution in git worktrees. Use when evaluating prompt improvement effects or learning prompt engineering through concrete examples.
llm-evaluation
LLM evaluation — automated metrics, human feedback, benchmarking. Use when testing performance, measuring AI quality, or establishing evaluation frameworks.
fetch-llm-papers
Refresh the LLM landscape paper pool (section/xllmpapers.md) with fetchllmpapers.py. USE FOR: refreshing citation counts or expanding topic coverage. DO NOT USE FOR: hand-curated entries (use add-new-entry-from-temp-md) or bestpractices.md citation counts (use update-cite-count).
ai-prompt-regression-testing
Prompt engineering regression test matrix register: baseline outputs, semantic drift thresholds, judge evaluations, and golden dataset.
ai-prompt-regression-testing
Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.