What is compare-embedding-models?

A comparison tool for testing two text-embedding models—the models that turn text into searchable numbers—on your own data or a sample fixture. It produces an HTML report with speed, storage, and result-quality findings.

Use it to compare indexing time, query delay, batch processing speed, storage use, and search quality between two embedding models.

Model leaderboards may not reflect your hardware or knowledge base. This gives you comparable evidence for choosing a model or recording why a change was worthwhile.

How jeanibarz/knowledge-base-mcp-server configures Claude Code

Reuse

Borrowing it

Nothing to install: this file belongs to jeanibarz/knowledge-base-mcp-server. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/jeanibarz/knowledge-base-mcp-server/main/.claude/skills/compare-embedding-models/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/jeanibarz/knowledge-base-mcp-server

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models/github.svg)](https://agentmods.dev/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models)

Your own site

<a href="https://agentmods.dev/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models"><img src="https://agentmods.dev/badge/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models"><img src="https://agentmods.dev/badge/skills/jeanibarz/knowledge-base-mcp-server/compare-embedding-models.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 45 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,605 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe. Third-party audits

NVIDIA SkillSpector pass 7 Sept 2026

How audits are shown

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00045	$0.01605
Opus 5.5	$0.00018	$0.00642
Sonnet 5.5	$0.00009	$0.00321
Haiku 4.5	$0.00005	$0.00161

Measured 2mo ago against content hash 02f3f114ec95, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

compare-embedding-models scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

| `OLLAMA_BASE_URL unreachable` | Ollama daemon not running | `ollama serve` in another terminal; verify with `curl $OLLAMA_BASE_URL/api/tags`. |

.claude/skills/compare-embedding-models/SKILL.md · 102 lines

How it starts

The opening of the file, as written. The whole thing — 102 lines — stays where its author put it; the contents beside it link to each section on GitHub.

When to use

The user is choosing between two embedding models for a new knowledge base and wants concrete numbers (latency, cost, storage, quality) on their hardware, not a generic leaderboard.
The user has switched models in the past and wants to verify the trade-off was worth it.
The user is documenting a model choice for a team and needs an HTML artefact to attach to a decision record / RFC / PR.

Prerequisites

knowledge-base-mcp-server 0.3.x installed (M0–M4 shipped; this skill needs kb models {add, list} from §4.4 of RFC 013).
Both models reachable: Ollama running locally (OLLAMA_BASE_URL) for ollama models; HUGGINGFACE_API_KEY set for HF models; OPENAI_API_KEY set for OpenAI models. The orchestrator reads provider tokens from env per src/config.ts.
Disk space: ~10 MiB per model index for the medium synthetic fixture; more for larger profiles.
For paid providers: estimated cost surfaced in the orchestrator preamble; non-zero requires --yes or interactive confirmation (src/cli.ts runAddModel flow, RFC 013 §4.4).

Steps

Identify model ids. kb models list shows registered models with their <provider>__<slug> ids. If a target model is not yet registered, run kb models add <provider> <model_name> first (the orchestrator can auto-register with --yes, but registering explicitly lets the user audit cost upfront).
Pick a fixture profile.
- --fixture=small (~150 chunks) — sanity check, ~10 s.
- --fixture=medium (~600 chunks) — default; ~1 min on Ollama, ~30 s on HF/OpenAI.
- --fixture=external — runs against the corpus at KNOWLEDGE_BASES_ROOT_DIR (the user's real KB; no copy is made).
- --fixture=large (~3000-chunk arxiv corpus) — selection-grade. Not yet implemented in v1; deferred to M5.1 (RFC 013 §4.13.4 follow-up).

Run the comparison:

npm run bench:compare -- \
  --models=ollama__nomic-embed-text-latest,huggingface__BAAI-bge-small-en-v1.5 \
  --fixture=medium \
  --concurrency=1,4,16

Read the full file on GitHub · 102 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 102 lines · 45 tokens per session scan A 02f3f114ec95

Subscribe to this mod's changes

compare-embedding-models is a skill published in the GitHub repository jeanibarz/knowledge-base-mcp-server (53 stars, last pushed yesterday), licensed Unlicense. It adds 45 tokens to every session and 1,605 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Analytics and metrics Data and AI benchmark comparison embeddings html-report latency models selection throughput

compare-embedding-models next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
compare-embedding-modelsjeanibarz/knowledge-base-mcp-server	Skill	53	yesterday	Unlicense	A	45	Claude Code
Inferenceagentic-in/elephant-agent	Skill	584	2mo ago	none	A	21	Claude Code, Codex
rag-perfPracticalSwan/agent-skills	Skill	24	18d ago	MIT	A	30	Codex
arrowspacesickn33/agentic-awesome-skills	Skill	49k	yesterday	MIT	A	28	Claude Code
autorag-setupMarker-Inc-Korea/AutoRAG	Skill	5.3k	yesterday	Apache-2.0	A	75	Claude Code, Codex
autorag-lite-setupMarker-Inc-Korea/AutoRAG	Skill	5.3k	yesterday	Apache-2.0	A	65	Claude Code, Codex
together-ai-inferencesynthetic-sciences/openscience	Skill	4.3k	2d ago	Apache-2.0	A	55	Claude Code, Codex

knowledge-base-mcp-server: Skill for Claude Code