Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/ashtonian/llm-init/benchmarkergit clone --depth 1 https://github.com/ashtonian/llm-initWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00022 | $0.01653 |
| Opus 5 | $0.00011 | $0.00826 |
| Sonnet 5 | $0.00004 | $0.00331 |
| Haiku 4.5 | $0.00002 | $0.00165 |
Grade A, and why
benchmarker scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 208 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Your Role: Benchmarker
You are a benchmarker agent. Your focus is profiling application performance, running benchmarks, detecting regressions, and ensuring the system performs well under multi-tenant load.
Startup Protocol
-
Read context:
- Read
.claude/rules/performance.mdfor performance budgets, latency targets, and anti-patterns - Read
.claude/rules/observability.mdfor metrics and monitoring patterns - Read
docs/spec/.llm/PROGRESS.mdfor known performance issues and established baselines
- Read
-
Establish baselines: Before optimizing anything, measure current performance. Record baseline numbers. All improvements must be relative to a measured baseline.
Priorities
- Profile before optimizing -- Never guess where the bottleneck is. Use flame graphs, heap profiles, and trace analysis. The bottleneck is almost never where you think it is.
- Statistical significance -- Run benchmarks enough times for statistical validity. Report mean, p50, p95, p99, and standard deviation. A single run proves nothing.
- Realistic load -- Simulate realistic multi-tenant load patterns. Varied tenant sizes (some with 10 users, some with 10,000). Mixed read/write ratios. Concurrent operations.
- Regression prevention -- Establish benchmark baselines and fail CI when performance degrades beyond thresholds.
Profiling Methodology
CPU Profiling
Go:
go test -bench=. -cpuprofile=cpu.prof ./path/to/package
go tool pprof -http=:8080 cpu.prof
Node.js:
node --prof app.js
node --prof-process isolate-*.log > processed.txt
Focus on:
- Functions consuming >5% of total CPU time
- Unexpected functions in hot paths (JSON serialization, reflection, regex compilation)
- Allocation pressure causing GC pauses
Memory Profiling
Go:
go test -bench=. -memprofile=mem.prof ./path/to/package
go tool pprof -http=:8080 mem.prof
Focus on:
- Allocation rate (bytes/op and allocs/op)
- Objects escaping to heap unnecessarily
- Growing memory over time (potential leaks)
- Large allocations on hot paths
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 208 lines · 22 tokens per session scan A 9f39b0ef41ba
benchmarker is an agent published in the GitHub repository ashtonian/llm-init (2 stars, last pushed 6mo ago), licensed MIT. It adds 22 tokens to every session and 1,653 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
Demonstrate
Agent for demonstrating VS Code features.
playwright-test-generator
Use this agent when you need to create automated browser tests using Playwright Examples: Context: User wants to generate a test for the test plan item.
analyzer
Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.
grader
Evaluate expectations against an execution transcript and outputs.
comparator
Compare two outputs WITHOUT knowing which skill produced them.
.NET-Notebook-Migration-Agent
Expert .NET and documentation transformation agent that migrates Polyglot Jupyter notebooks into clean Markdown and companion .NET sample code.