Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/rbraga01/builder-ai/model-benchmarkingnpx skills add RBraga01/builder-ai --skill model-benchmarkinggit clone --depth 1 https://github.com/RBraga01/builder-aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/rbraga01/builder-ai/model-benchmarking)<a href="https://agentmods.dev/skills/rbraga01/builder-ai/model-benchmarking"><img src="https://agentmods.dev/badge/skills/rbraga01/builder-ai/model-benchmarking.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00043 | $0.01254 |
| Opus 5 | $0.00022 | $0.00627 |
| Sonnet 5 | $0.00009 | $0.00251 |
| Haiku 4.5 | $0.00004 | $0.00125 |
Grade A, and why
model-benchmarking scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 121 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Model Benchmarking
The Law
A MODEL IS NOT CHOSEN UNTIL IT HAS BEEN TESTED ON YOUR TASK DATA.
Leaderboard scores are averaged over tasks you're not building.
"It's the best model" is a claim about benchmarks someone else ran.
Task data + defined metric + three tested tiers IS a model selection.
When to Use
Trigger when:
- Choosing a model for any new production feature
- Considering a switch to a newer, cheaper, or faster model
- Validating that a smaller model can replace a larger one
- Comparing providers (OpenAI, Anthropic, Mistral, local)
When NOT to Use
- Initial feasibility exploration (before task definition is stable) — benchmark when you know what you're measuring
- Model compatibility checks (does this model support tool use, JSON mode, etc.) — that's a capability query, not a benchmark
- Leaderboard research to narrow the candidate list — that's input to Stage 3, not a substitute for it
The Process
Four stages. Do not collapse them.
Stage 1 — Define the Bar
Before running any model, write down:
| Decision | What to Define |
|---|---|
| Task | Exact input format, exact output format, edge cases |
| Metric | Accuracy, faithfulness score, extraction F1, LLM-as-judge, task pass rate |
| Pass threshold | The minimum score to go to production (e.g., ≥ 88%) |
| Budget constraint | Max cost per 1k calls, max monthly spend |
| Latency constraint | Max acceptable p95 (e.g., ≤ 2.0s) |
If you cannot define the metric and threshold first, the selection criteria are not clear enough to proceed.
Stage 2 — Build a Representative Test Set
- Minimum 50 examples; 200+ for features used at volume
- Label ground truth before running any model. Seeing outputs first contaminates labels.
- Include: easy cases, hard cases, edge cases, adversarial inputs
- Tag each example by difficulty:
easy / medium / hard - Store in
benchmarks/<feature>/test-set.jsonl
Stage 3 — Benchmark at Least Three Tiers
Always test at least one tier below your expected choice:
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 121 lines · 43 tokens per session scan A 7cd95fd00299
model-benchmarking is a skill published in the GitHub repository RBraga01/builder-ai (2 stars, last pushed 1mo ago), licensed MIT. It adds 43 tokens to every session and 1,254 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.
promptfoo-evals
Write, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for adversarial redteam plugin or strategy…
prompt-master
Generates optimized prompts for AI tools. Activates only when the user explicitly asks to write, fix, improve, or adapt a prompt for a specific AI tool (LLM, Cursor, Midjourney, image AI, video AI, coding agents, etc.). Does not activate for general conversation, coding tasks, document writing, or other…
flux2-lora-training
Comprehensive reference for training LoRAs on FLUX.2 Klein 9B and Qwen Image Edit 2511 models. Use this skill whenever the user asks about: training LoRAs for flux2/flux 2 klein/qwen-image-edit, before/after edit LoRAs (head swap, face swap, image editing), inpainting LoRAs, training at larger resolutions, latent…
deepseek-provider-contract
Validate a proposed DeepSeek API integration before any key or project context is sent: check thinking-mode tool-call history, strict-schema assumptions, bounded output, and provider data boundaries. Use when integrating DeepSeek, adding DeepSeek tool calls or streaming, debugging DeepSeek 400 after a tool call, or…
forensic-prompt-compiler
Forensic image-to-prompt compiler for image generation models. Use this skill whenever the user wants to: convert/describe an existing image into a generation prompt, reconstruct a scene as a prompt, generate prompts from reference images for AI image tools (Midjourney, FLUX, Stable Diffusion, DALL-E, or any diffusion…