A benchmark-run skill for evaluating local language models against a temporary Harbor Clerk setup. A benchmark is a repeatable test used to compare software or models under defined conditions.

Use it only when asked to run the evaluation sweep, recheck models after a llama.cpp update, or gather evidence for promoting a model.

It provides a dated report and pull request for a model comparison, while checking that the machine is suitable before spending time and money. The run can erase the test instance and takes hours.

How r0shi/harborclerk configures Claude Code

Reuse

Borrowing it

Nothing to install: this file belongs to r0shi/harborclerk. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/r0shi/harborclerk/main/.claude/skills/benchmark/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/r0shi/harborclerk

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/r0shi/harborclerk/benchmark/github.svg)](https://agentmods.dev/skills/r0shi/harborclerk/benchmark)

Your own site

<a href="https://agentmods.dev/skills/r0shi/harborclerk/benchmark"><img src="https://agentmods.dev/badge/skills/r0shi/harborclerk/benchmark/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/r0shi/harborclerk/benchmark"><img src="https://agentmods.dev/badge/skills/r0shi/harborclerk/benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 73 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 2,848 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00073	$0.02848
Opus 5.5	$0.00029	$0.01139
Sonnet 5.5	$0.00015	$0.00570
Haiku 4.5	$0.00007	$0.00285

Measured 21d ago against content hash 75bd52eee7b8, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

benchmark scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 21d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

curl -s http://localhost:8100/api/system/health | jq '{status, build}'

.claude/skills/benchmark/SKILL.md · 196 lines

How it starts

The opening of the file, as written. The whole thing — 196 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Benchmark run

Drives scripts/test_corpora (README and RUNBOOK there) and turns the run into docs/reports/YYYY-MM-DD-benchmark-<host>-<run>.md and a PR. All commands run from the repository root. A run wipes the instance it targets, spends money, and takes hours. Start one only when the owner asked for it, and say what it will cost in time and dollars before it starts.

1. Is the machine fit to measure on?

RUN=bench-$(date +%Y%m%d-%H%M)
WORKDIR="$HOME/Library/Application Support/Harbor Clerk/test-corpora"
MODELS=qwen3-8b,qwen35-9b          # what the owner asked to compare
uv --project scripts/test_corpora run python -m scripts.test_corpora.preflight \
  --models "$MODELS" --json "$WORKDIR/results/$RUN/preflight.json"

It checks thermal pressure, power, free memory, a busy GPU at idle, other model servers (Ollama), that the installed llama-server is at the pin, that the spaCy model the metrics load is installed (a fresh venv lacks it; the check names the install command and runs whatever phases the run will do, #683), that each model fits this machine, and that the instance is healthy. It changes nothing.

fail stops the run. A number taken on a throttled machine, or through a stale llama-server, is not a baseline, and a report that says so is still not evidence for a model decision. Tell the owner what failed; the fixes are theirs (moving the machine, make apps, launchctl bootout for a launch agent). The mini sat at thermal pressure "Sleeping" for weeks before this check existed (#652).
warn travels with the report. Say it in the Reading. A check that could not look is a warn, never a pass.
If the app has a model loaded, note which (GET /api/chat/models/status; "Afterwards" restores it), then deactivate it (Models page, or PUT /api/chat/models/deactivate): a resident 22 GB model fails the free-memory check, and the sweep activates what it needs.
Never launch a llama-server by hand for this. The sweep drives the app's own server through the API, and the app clamps context to what fits (macos/AGENTS.md, "A model server does not run out of memory").

Read the full file on GitHub · 196 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

21d ago Changed · +26 lines 75bd52eee7b8
1mo ago Changed · +5 lines cc71f39a0b99
1mo ago Changed · +2 lines 1ae7c94597d1
1mo ago Changed · +10 lines 34dd5511139f
1mo ago Changed · +3 lines f69ab1853868
1mo ago First seen · 150 lines · 73 tokens per session scan A c770b41655fa

Subscribe to this mod's changes

benchmark is a skill published in the GitHub repository r0shi/harborclerk (3 stars, last pushed 2d ago), licensed MIT. It adds 73 tokens to every session and 2,848 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-20.

Analytics and metrics Data and AI LLM docker document-intelligence hybrid-search local-first macos

benchmark next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
benchmarkr0shi/harborclerk	Skill	3	2d ago	MIT	A	73	Claude Code
memobasemaximskorohod/memobase	Skill	22	17d ago	MIT	A	25	Claude Code, Codex
run-llmsav/harbor	Skill	3.3k	6d ago	Apache-2.0	A	114	Claude Code, Codex
harborav/harbor	Skill	3.3k	6d ago	Apache-2.0	A	114	Claude Code
rag-knowledgevstorm-co/full-stack-ai-agent-template	Skill	2.0k	7d ago	MIT	A	76	Claude Code
gnogmickel/gno	Skill	117	4d ago	MIT	A	86	Claude Code
difyznlgis/opengis-skills	Skill	76	10d ago	MIT	A	115	Claude Code, Codex

harborclerk: Skill for Claude Code