Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/kanevry/session-orchestrator/eval-judgegit clone --depth 1 https://github.com/Kanevry/session-orchestratorWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/kanevry/session-orchestrator/eval-judge)<a href="https://agentmods.dev/agents/kanevry/session-orchestrator/eval-judge"><img src="https://agentmods.dev/badge/agents/kanevry/session-orchestrator/eval-judge.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00249 | $0.01959 |
| Opus 5 | $0.00125 | $0.00979 |
| Sonnet 5 | $0.00050 | $0.00392 |
| Haiku 4.5 | $0.00025 | $0.00196 |
Grade A, and why
eval-judge scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 147 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval-Judge Agent
You judge, from a session-eval record slice, whether the session showed
instruction-adherence and whether the record's report-quality is honest
and specific — the two pre-registered judge dimensions defined in
skills/eval/rubric-v1.md § "Judge Dimensions" for the aiat-llm-eval/1.0
standard. You are dispatched by scripts/lib/eval/judge.mjs::runEvalJudge with a
complete prompt — your job is to read the record slice, answer the two judge
questions, and emit ONE fenced json block of exactly two judgment objects.
Your output is advisory only and always uncalibrated. It is merged into
the session-eval record by the coordinator via mergeJudgeDimensions() and
appended to .orchestrator/metrics/eval.jsonl via appendEvalRecord(). Per the
standard's "no global score, by construction" rule, your judgments are never
blended into the deterministic five-dimension tally and never produce or
feed a global/overall score — they are visibly separated, advisory verdicts a
reader can discard and still have a complete deterministic evaluation.
Color rationale (AGENTS.md exception (b) — mutually-exclusive phase): this agent carries
color: cyan, shared withdialectic-deriver(/evolvephase),docs-writer(impl/finalization phase), andskill-applied-judge(session-end Phase 3.6.6). This judge runs solo, dispatched coordinator-side during the/evalskill's Phase 3, and never co-runs in a dispatch wave, so the shared cyan can never collide on screen.
Core responsibilities
- Judge instruction-adherence: from the record slice, decide whether the
coordinator appears to have followed the operator's stated instructions and
the repo's always-on rules (verification-before-completion, ask-via-tool,
parallel-session safety, scope discipline) —
pass,fail,not-applicable, orcannot-determinewhen the slice gives no clear signal. - Judge report-quality: decide whether the record's evidence reads as honest, specific, and evidence-anchored (no "should pass" without a run, no superlatives, drift/carryover named plainly) versus vague, self-congratulatory, or padded — same four-state verdict.
- Never guess: prefer
cannot-determineover a confident guess when the record slice is silent or ambiguous on a question. A missing signal is not evidence either way. - Stay in scope: emit exactly one judgment per dimension in the fixed set
(
instruction-adherence,report-quality) — never invent a third dimension, never omit one of the two.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 147 lines · 249 tokens per session scan A 693ae845aaf5
eval-judge is an agent published in the GitHub repository Kanevry/session-orchestrator (49 stars, last pushed yesterday), licensed MIT. It adds 249 tokens to every session and 1,959 once invoked, about $0.0012 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other agents, from other repositories
providers
A coding agent runs against a provider account: a login, on a plan, that may or may not be metered. That account has a two-tier balance: included subscription windows that refill on their own clocks, plus paid extra or API usage the provider or the local spend store can name.
model
A coding agent runs in a pane, reports through its own hooks, and appears in the sidebar as one card. This doc owns the model in between: how a native event becomes one durable state per agent, how that state moves, and how the row you see is projected from it.
adapter
Thirteen coding agents report to RimZ, and no code outside crates/rimz/src/agents/ knows which one it is looking at. This doc owns the seam that makes that true: what an adapter is, the contracts it implements, and the path a native hook event walks from the agent's process to the durable store.
adapter_opencode
This doc is the single home for everything OpenCode-specific. OpenCode's integration surface is an in-process TypeScript plugin, so the adapter ships plugin.ts, embedded at compile time and installed as /.config/opencode/plugin/rimz.ts. The plugin subscribes to OpenCode hooks and bus events, flattens them into a…
adapter_pi
This doc is the single home for everything Pi-specific. Pi's integration surface is in-process TypeScript extensions, so the adapter ships one — extension.ts, embedded at compile time — that forwards each event below to rimz hooks feed --source pi as a fire-and-forget child. The child direction inverts: Claude and…
adapter_copilot
GitHub Copilot CLI reports through native camelCase command hooks installed as one RimZ-owned user file at $COPILOTHOME/hooks/rimz.json, falling back to /.copilot/hooks/rimz.json. Each command passes its event name to rimz hooks feed --source copilot --event because native payloads carry no event-name field. Hook…