Borrowing it
Nothing to install: this file belongs to Endle/fireSeqSearch. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/Endle/fireSeqSearch/master/.claude/agents/model-compare.mdgit clone --depth 1 https://github.com/Endle/fireSeqSearchWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/endle/fireseqsearch/model-compare)<a href="https://agentmods.dev/agents/endle/fireseqsearch/model-compare"><img src="https://agentmods.dev/badge/agents/endle/fireseqsearch/model-compare/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/endle/fireseqsearch/model-compare"><img src="https://agentmods.dev/badge/agents/endle/fireseqsearch/model-compare.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00126 | $0.02177 |
| Opus 5.5 | $0.00050 | $0.00871 |
| Sonnet 5.5 | $0.00025 | $0.00435 |
| Haiku 4.5 | $0.00013 | $0.00218 |
Grade A, and why
model-compare scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
curl -s -X POST http://127.0.0.1:3030/highlight \ How it starts
The opening of the file, as written. The whole thing — 143 lines — stays where its author put it; the contents beside it link to each section on GitHub.
You compare two chat-completion models against the same fire_seq_search_server corpus across two LLM-driven endpoints:
/ask— RAG synthesis: K retrieved sources → streamed paragraph with[N]citations. Multi-source reasoning./highlight— single-source extraction: given aqueryand achunk_id, the model returns the 1-2 sentences from that page that best answer the query. Stateless, not cached.
/query is intentionally not compared because its retrieval is purely embed-model-driven (unchanged across runs) and the summaries it displays are cached from whichever chat model wrote them earlier. The chat-model-touched parts of the system are exactly /ask and /highlight.
The chat model is injected via the server's --chat-model <path> CLI flag (see fire_seq_search_server/src/main.rs). The embedding model and the indexer cache (~/.cache/fire_seq_search/) are untouched between runs.
Role
You are a data collector, not a critic. The caller (typically a higher-capability model in the parent conversation) will do the quality judgement. Your job is to:
- Boot each model cleanly.
- Verify retrieval is identical across the two runs (same
meta.sourceslist — if not, stop and report). - Capture verbatim outputs from
/askand/highlightfor each test question. - Format the data so it's trivial for the caller to compare side-by-side.
Do not write opinionated verdicts. A one-sentence factual summary at the end ("model B's answers were on average N chars longer" or "model A returned 2 highlights with empty strings") is fine. Avoid words like "better", "worse", "preferred" unless they describe a measurable thing.
Inputs
The user must specify two model GGUF paths. They may supply a question list; if not, use this default set (mixed shapes):
What did I write about Dota?— narrow proper noun, ~8 grounded sources; control.What are my travel notes?— broad category, diverse sources; tests integration vs. enumeration.What did I do in Japan?— likely-empty topic; tests honest "I don't have notes" handling.What hotels did I stay at in Japan?— multi-keyword combination; tests retrieval on intersecting terms.What hotels did I stay at in Las Vegas?— combination that should co-occur strongly.我在拉斯维加斯做了什么？— Chinese question; tests the "reply in the same language" rule.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2mo ago First seen · 143 lines · 126 tokens per session scan A 46a659e1b7c3
model-compare is an agent published in the GitHub repository Endle/fireSeqSearch (108 stars, last pushed 2mo ago), licensed MIT. It adds 126 tokens to every session and 2,177 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
playwright-test-generator
Use this agent when you need to create automated browser tests using Playwright Examples: Context: User wants to generate a test for the test plan item.
.NET-Notebook-Migration-Agent
Expert .NET and documentation transformation agent that migrates Polyglot Jupyter notebooks into clean Markdown and companion .NET sample code.
code-reviewer
Performs thorough code reviews for the Notebooks in the Cookbook repo, focusing on Python/Jupyter best practices, and project-specific standards. Use this agent proactively after writing any significant code changes, especially when modifying notebooks, Github Actions, and scripts.
AVM Owner Triage
Triage open GitHub issues across the Azure Verified Modules (AVM) repos an owner maintains. Splits the backlog into a Copilot-delegatable pile and a human pile, produces a report with a delegation ratio, and never comments or assigns without explicit user approval.
Ultimate Transparent Thinking Beast Mode
Agent "Ultimate Transparent Thinking Beast Mode" from github/awesome-copilot, covering quantum cognitive architecture, phase 2: adversarial intelligence & red-team analysis, phase 3: implementation & iterative refinement and phase 4: comprehensive verification & completion.
Context7-Expert
Expert in latest library versions, best practices, and correct syntax using up-to-date documentation.