Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/archive228/loopkit/eval-harnessnpx skills add Archive228/loopkit --skill eval-harnessgit clone --depth 1 https://github.com/Archive228/loopkitWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00054 | $0.00867 |
| Opus 5 | $0.00027 | $0.00434 |
| Sonnet 5 | $0.00011 | $0.00173 |
| Haiku 4.5 | $0.00005 | $0.00087 |
Grade A, and why
eval-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 73 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Harness
Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships .claude/agents/verifier.md — that is your grader. Do not rebuild it.
The three-stage loop
inputs.jsonl → runner → outputs.jsonl → verifier (per row) → verdicts.jsonl → diff vs baseline
Each stage writes to disk. No stage holds the whole run in context.
Stage 1 — inputs.jsonl
One JSON object per row: {"id": "case-01", "input": "...", "expected": "..."}.
- 20-100 cases is enough for a signal. More is nice, not required.
- Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
- Freeze the file. Rev the eval with a suffix (
inputs-v2.jsonl) when you change it. Never edit in place — you lose the baseline.
Stage 2 — runner
A dumb loop: for each row, call the model with the current prompt/skill, capture output, write {"id": ..., "output": ...} to outputs.jsonl. No grading here — just capture.
- Same temperature every run (usually 0 for evals).
- Same seed / model version.
- Log the git SHA of the prompt/skill under test in the file header.
If the runner is smart it will bias the eval. Keep it dumb.
Stage 3 — verifier
Fan out one subagent per row (see subagent-fanout). Each gets:
- The input.
- The expected output (or spec).
- The actual output.
- The verifier system prompt from
.claude/agents/verifier.md.
Verifier returns strict JSON: {"pass": bool, "why": "..."}. Collect into verdicts.jsonl.
Diff vs baseline
Two runs of the same eval on two prompt versions → compare pass rates per case. What matters:
- Overall pass rate — the headline.
- Regressions — cases that were green and went red. These block ship.
- New passes — cases that were red and went green. These justify ship.
- Flappy cases — inconsistent across reruns. Investigate; may be genuine model nondeterminism or a bad case.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 73 lines · 54 tokens per session scan A 4110bc927e29
eval-harness is a skill published in the GitHub repository Archive228/loopkit (753 stars, last pushed 1mo ago), licensed MIT. It adds 54 tokens to every session and 867 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
weekly-digests
Generate a serial week-by-week narrative digest of a project's full claude-mem timeline. Splits the timeline into per-ISO-week files, then runs one consecutive subagent per week — each receiving the prior week's carry-forward block — to produce one chapter per ISO week of data. Use when asked for "weekly digests"…
agentmail
Use when an agent needs AgentMail CLI email inboxes.
skill-template
Template for creating new Agent Skills for context engineering. Use this template when adding new skills to the collection.
Agent Development
This skill should be used when the user asks to "create an agent", "add an agent", "write a subagent", "agent frontmatter", "when to use description", "agent examples", "agent tools", "agent colors", "autonomous agent", or needs guidance on agent structure, system prompts, triggering conditions, or agent development…
skill-creator
Scaffold a new yoyo skill when a human or community issue asks for one ("add a skill for X", "create a skill that does Y"). Generates correct frontmatter, validates, writes to disk.
subagent-delegation
Canonical protocol for delegating GSD work to native Antigravity subagents — when to delegate, how to invoke, workspace isolation modes, and the inline fallback for older IDE versions.