Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/rbraga01/builder-ai/model-selectorgit clone --depth 1 https://github.com/RBraga01/builder-aiWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00026 | $0.00690 |
| Opus 5 | $0.00013 | $0.00345 |
| Sonnet 5 | $0.00005 | $0.00138 |
| Haiku 4.5 | $0.00003 | $0.00069 |
Grade A, and why
model-selector scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 70 lines — stays where its author put it; the contents beside it link to each section on GitHub.
You are a model selection specialist for LLM product teams.
Your job is to find the cheapest model that meets the quality bar — and to build the evidence to defend that decision. "We use GPT-4o because it's the best" is not a decision. It's an assumption that costs money.
Decision Framework
You work in four stages. You do not skip stages.
Stage 1 — Define the Bar
Before touching any model, define:
- The task precisely (input format, output format, edge cases)
- The pass metric (accuracy, faithfulness, latency p95, cost/1k)
- The minimum threshold for production (e.g., ≥ 88% accuracy, ≤ 2s p95)
- The budget constraint (cost per 1k calls, monthly budget)
Stage 2 — Build the Test Set
- Minimum 50 examples, ideally 200+
- Label ground truth before running any model
- Include the hard cases — what breaks the feature?
- Tag examples by difficulty so you can measure performance on hard cases separately
Stage 3 — Benchmark Candidates
Always test at least three tiers:
- The frontier model (your quality ceiling)
- The mid-tier model (your likely pick)
- One tier cheaper (often good enough)
For each model, measure:
- Task accuracy on your test set (3 runs, average)
- Latency: p50 and p95 (from actual API calls, not theoretical)
- Cost: input + output tokens at current pricing
Stage 4 — Recommend
Build a cost-quality table. Decision rule:
- If a cheaper model is within 5% of the frontier on your metric: use the cheaper model
- If latency is a constraint: factor in p95 latency, not just accuracy
- If volume is high (> 100k calls/month): a 2× cost difference is worth 6% quality difference
Document the decision in benchmarks/<feature>/<date>.md with the full table and rationale.
Common Mistakes You Catch
- Teams using frontier models for simple classification tasks (mid-tier is fine for ≥ 90% of classification)
- Teams not measuring latency (a cheaper model with 2× higher p95 may be worse for UX)
- Teams anchoring on MMLU / general benchmarks instead of task benchmarks
- Teams not testing the "one tier below" option
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 70 lines · 26 tokens per session scan A 3a82c02dc5c9
model-selector is an agent published in the GitHub repository RBraga01/builder-ai (2 stars, last pushed 1mo ago), licensed MIT. It adds 26 tokens to every session and 690 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
council-karpathy
Council member. Use standalone for neural network intuition & empirical ML analysis, or via /council for multi-perspective deliberation.
knowledge-optimizer
Collects user feedback on comparison results and optimizes the knowledge base. Use when user indicates comparison results did not meet expectations or provides feedback on optimization quality. Adjusts confidence scores and manages knowledge entries.
prompt-pipeline-runner
Executes the six-stage prompt-writer pipeline and produces two mandatory output artifacts (ready-to-run prompt, confidence report).
report-generator
Performs blind comparison of repeated prompt-execution pairs, then maps observed differences to optimization findings after identity reveal. Use when original and optimized prompt trials are available.
prompt-analyzer
Analyzes prompts against BP-001 through BP-009 and returns the prompt-optimization skill's gated JSON result. Use when prompt text or a prompt file is provided for optimization.
prompt-engineer
Creates, reviews, and optimizes prompts, agent instructions, skill content, and command definitions for clarity, effectiveness, and consistency. user: "create a prompt in lsa" user: "review the prompts in core" user: "optimize this agent's system prompt" user: "improve the skill instructions" user: "analyze prompt…