Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/archive228/loopkit/evaluator-calibrationnpx skills add Archive228/loopkit --skill evaluator-calibrationgit clone --depth 1 https://github.com/Archive228/loopkitWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00028 | $0.00931 |
| Opus 5 | $0.00014 | $0.00465 |
| Sonnet 5 | $0.00006 | $0.00186 |
| Haiku 4.5 | $0.00003 | $0.00093 |
Grade A, and why
evaluator-calibration scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 52 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Evaluator Calibration
An evaluator agent that reads the generator's reasoning drifts lenient. The generator explains why the code is good; the evaluator, priming on that prose, starts nodding along. By sprint 8 the "skeptical critic" is a rubber stamp. Prithvi flagged this in the March 2026 planner/generator/evaluator writeup — evaluator leniency is the failure mode of the three-agent harness.
The fix is not "tell the evaluator to be stricter." That works for one iteration. The fix is anchoring the rubric with concrete pass/fail examples the evaluator re-reads every invocation, and re-prompting from scratch on a fixed cadence so drift can't accumulate.
When to apply
- You're building a critic/evaluator/judge agent in a multi-agent loop.
- You're using an LLM as a grader for skills, PRs, diffs, or agent output.
- You've noticed pass rates creeping up while output quality hasn't changed — or worse, dropped.
- You want two runs of the same evaluator on the same artifact to return the same verdict.
Procedure
-
Write the rubric as a scored checklist, not prose. Each criterion gets a name, a one-line definition, and a binary or 1-3 score. Prose rubrics ("evaluate whether the code is well-designed") drift; checklists don't.
-
Anchor every criterion with 2 concrete examples — one pass, one fail. Real examples from prior runs, not invented ones. The evaluator reads these every invocation. This is the calibration; without it you're just prompting hope.
-
Forbid reading the generator's reasoning before scoring. The evaluator sees the artifact (code, diff, output) and the rubric. It does not see the generator's "here's why this is good" prose. Score first, then optionally read the reasoning to write the critique.
-
Require the evaluator to quote the artifact in every verdict. "Fails criterion 3 because " — not "fails criterion 3." Quoting forces grounding and makes the verdict auditable.
-
Re-prompt from scratch every N iterations. Empirically N=5 works. Kill the evaluator's context, reload the system prompt + rubric + examples fresh. Do not compact; compaction preserves the drift.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 52 lines · 28 tokens per session scan A add92534b458
evaluator-calibration is a skill published in the GitHub repository Archive228/loopkit (753 stars, last pushed 1mo ago), licensed MIT. It adds 28 tokens to every session and 931 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
omni-mcp
Connect to the OmniRoute MCP server (110 tools, 3 transports: SSE/stdio/HTTP). Covers routing, cache, compression, memory, skills, providers, and audit tools across 33 permission scopes.
add-resend
Add Resend (email) channel integration via Chat SDK.
agui-dotnet-sample-step
Add a GettingStarted sample Step (a Server/Client pair) to the AG-UI .NET SDK that demonstrates one protocol feature the way we want users to write it. USE FOR: adding a new samples/GettingStarted/StepNN Server+Client pair, wiring it into AGUI.slnx and the integration-test project, giving it a deterministic…
a-evolve
Apply A-Evolve's agentic evolution methodology to improve AI agent performance across runs. Use when the user wants to diagnose agent failures, generate targeted skills from error patterns, evolve system prompts, or accumulate episodic knowledge. Works standalone or inside AutoResearchClaw pipelines. Triggers on…
workflows
Execute a workflow script that orchestrates multiple subagents deterministically. Use when the user explicitly requests a workflow, multi-agent orchestration, parallel or sequential agent pipelines, structured agent outputs, or a durable background workflow run.
agent-tuning
Use when changing agent model or effort configuration, adapter mappings, or eval candidate profiles.