What is model-compare?

A blind comparison tool that sends the same prompt to two to four AI models and shows their answers anonymously. After you choose a response, it reveals which model produced it.

Use it for side-by-side prompt tests, model selection, prompt-writing experiments, custom judging criteria, answer synthesis, and recording comparison votes.

It helps compare models without being influenced by their names and makes it easier to choose a model for a particular task or workflow.

model-compare — Skill for Codex by moonlight-lupin

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add moonlight-lupin/agent-skills --skill model-compare

Clone the repo

git clone --depth 1 https://github.com/moonlight-lupin/agent-skills

Made for: Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/moonlight-lupin/agent-skills/model-compare/github.svg)](https://agentmods.dev/skills/moonlight-lupin/agent-skills/model-compare)

Your own site

<a href="https://agentmods.dev/skills/moonlight-lupin/agent-skills/model-compare"><img src="https://agentmods.dev/badge/skills/moonlight-lupin/agent-skills/model-compare/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/moonlight-lupin/agent-skills/model-compare"><img src="https://agentmods.dev/badge/skills/moonlight-lupin/agent-skills/model-compare.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 145 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 9,453 The whole file, excluding the scripts and references it only reads on demand.

Security scan C 3 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00145	$0.09453
Opus 5	$0.00072	$0.04727
Sonnet 5	$0.00029	$0.01891
Haiku 4.5	$0.00015	$0.00945

Measured 9d ago against content hash 497568503a08, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-09, from the pricing page.

Security

Grade C, and why

model-compare scanned grade C with 3 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

The scan reads SKILL.md. This mod also ships 7 executable files (scripts/compare.py, scripts/embedding_compare.py, scripts/provider_health.py, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Downloads and executes remote codehighSupply chain

curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.

curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data'] if 'KEYWORD' in m['id'].lower()]"

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

The primary interface — a standalone CLI tool (no pip dependencies, pure stdlib + urllib). See `references/provider-tool-support.md` for which models support tool calling. For empirical model behavior findings from 16 he

Runs shell commandslowCapability

Expected in a hook, worth knowing in a rule or an instructions file.

- **Sandbox Python execution is not network-isolated** — the `run_python` tool runs `subprocess.run([sys.executable, script])` which can make network calls if the code imports `urllib` or `socket`. The 10-second timeout

mlops/model-compare/SKILL.md · 561 lines

How it starts

The opening of the file, as written. The whole thing — 561 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Send one prompt to multiple models simultaneously, present responses anonymously, let the user pick a winner, then reveal which model is which.

Inspired by the Compare feature in PewDiePie's Odysseus project, adapted for Hermes's multi-provider architecture (OpenRouter, NVIDIA, Ollama Cloud, any OpenAI-compatible endpoint).

When to use

"Compare these models on..."
"Which model is better for [task]?"
"A/B test [model1] vs [model2]"
"Blind comparison of..."
"Test how different models handle this prompt"
User wants to evaluate models before committing to one for a workflow
Prompt engineering — seeing how different models interpret instructions

When NOT to use

Benchmarking (MMLU, GSM8K, etc.) → use evaluating-llms-harness skill
Cost analysis → just check provider pricing pages
Single model test → just switch model and ask directly
Multi-source research synthesis → use deep-research skill (iterative research loop, not model comparison)

Architecture

Four comparison modes, all driven by scripts/compare.py:

Mode	Flag	What it does	API feature
simple	`--mode simple` (default)	One prompt → one response	Basic chat completion
tools	`--mode tools`	Multi-turn tool calling with real web_search/web_extract + sandboxed run_python/read_file/write_file. 10-turn max (configurable per-test via `max_turns` in TEST_BANK). Tracks full trace.	`tools` array in request, multi-turn messages
coding	`--mode coding`	Test bank coding prompts (LRU cache, concurrent fetch, debug merge sort, retry decorator)	Basic chat completion
review	`--mode review`	Code review prompts with planted bugs (SQL injection, clean code, race condition, float-for-money)	Basic chat completion

User prompt + model list
  → Step 1: Resolve models to provider endpoints (free providers first)
  → Step 2: Send prompt to all models in parallel
     ├─ simple/coding/review: one-shot chat completion
     └─ tools: multi-turn loop (Think→Search→Extract→Execute→Synthesize→Stop, max 10 turns by default, per-test configurable)
  → Step 3: Quality check responses (handle errors/empty)
  → Step 4: Present anonymously (shuffle + label A/B/C/D)
  → Step 5: Efficiency table (tokens in/out, turns, tool calls — auto for tools mode)
  → Step 6: User votes OR judge model evaluates
  → Step 7: Reveal identities + show mapping
  → Step 8 (optional): Save to JSON file

Read the full file on GitHub · 561 lines

Files

What ships with it

11 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

9d ago First seen · 561 lines · 145 tokens per session scan C 497568503a08

Subscribe to this mod's changes

model-compare is a skill published in the GitHub repository moonlight-lupin/agent-skills (57 stars, last pushed 2d ago), licensed MIT. It adds 145 tokens to every session and 9,453 once invoked, about $0.0007 per session on Opus 5. A static security scan graded it C with 3 findings (downloads and executes remote code, makes network calls, runs shell commands). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Building agents Claude Code agent-ops ai-agent ai-agent-skills automation hermes

SoloFlow

A meta-skill that silently watches your workflows and automatically generates reusable Hermes skills from them.

SonicBotMan/SoloFlow · 21 tokens

hermes-tweet

Use Xquik in Hermes Agent for public X research, monitoring, thread summaries, creator discovery & approved actions. Not affiliated with X Corp. Use when the user requests X data or a named action. Trigger with "search X", "monitor X", "post tweet", or "X trends".

Xquik-dev/hermes-tweet · 65 tokens

model-compare