Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Calibre-Labs/reforge-ai-evals --skill eval-rubricgit clone --depth 1 https://github.com/Calibre-Labs/reforge-ai-evalsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/eval-rubric)<a href="https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/eval-rubric"><img src="https://agentmods.dev/badge/skills/calibre-labs/reforge-ai-evals/eval-rubric/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/eval-rubric"><img src="https://agentmods.dev/badge/skills/calibre-labs/reforge-ai-evals/eval-rubric.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00148 | $0.02156 |
| Opus 5.5 | $0.00059 | $0.00862 |
| Sonnet 5.5 | $0.00030 | $0.00431 |
| Haiku 4.5 | $0.00015 | $0.00216 |
Grade A, and why
eval-rubric scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 183 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval rubric skill
An eval rubric is the written definition of good behavior for one AI product. It is a section of the PRD. Every line is a pass/fail check on a single trace. Numbers that hold across many traces (cost, latency, failure rate) are release thresholds and live in their own PRD section.
This skill writes v1 before any traces exist, and v2 after traces have been read. It also audits a rubric someone else wrote.
Read references/worked-examples.md before writing a rubric for the first time in a session. It
shows two products taken from v1 to v2.
Pick the mode
| The user has | Mode |
|---|---|
| A PRD, spec, system prompt or a product description, and no traces yet | v1 |
| A v1 rubric plus trace codes or annotated traces | v2 |
| A rubric and wants feedback on it | Audit |
If it is unclear, ask one question: "Have you read traces from this agent yet?"
The four kinds of lines
| Group | Question it answers | Comes from | Typical owner |
|---|---|---|---|
| Outcome (O) | What did it produce, and is it right? | Definition of success | PM, subject expert, engineering |
| Trajectory (T) | How did it get there? | Autonomy table, known risks | Engineering |
| Governance (G) | What must it never do, and always do? | Tool surface, policy, non-goals | Legal, finance, security |
| Experience (X) | How did it feel to the user in this channel? | User and channel | Design |
Rules for every line:
- One behavior per line. If a line has "and" joining two behaviors, split it.
- Pass or fail on one trace. A reviewer reading a single session can decide.
- Observable. The evidence is in the trace: an output, a tool call, an argument, a message.
- About the product, not one demo task. A line written for a coffee order should also hold for the pharmacy pickup. If it mentions the demo's subject, rewrite it.
- Specific to this product. Generic lines such as "is helpful" or "is safe" are not rubric lines.
- Governance lines start with "Never" or "Always".
- Give each line an ID: O1, O2, T1, G1, X1. IDs are stable across versions.
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 183 lines · 148 tokens per session scan A ef0dfa31b827
eval-rubric is a skill published in the GitHub repository Calibre-Labs/reforge-ai-evals (75 stars, last pushed yesterday), licensed MIT. It adds 148 tokens to every session and 2,156 once invoked, about $0.0006 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-10-17.
Other skills, from other repositories
agent-orchestration-advisor
Design multi-agent AI workflows with clear boundaries, handoffs, and monitoring. Use when a complex PM task should run as parallel specialized agents instead of one linear process.
ai-shaped-readiness-advisor
Assess whether your product work is AI-first or AI-shaped. Use when evaluating AI maturity and choosing the next team capability to build.
company-intel
Research a company, industry, or competitor set using web search and seven analytical lenses. Use when you need structured intel that feeds downstream PM skills.
context-engineering-advisor
Diagnose context stuffing vs. context engineering. Use when an AI workflow feels bloated, brittle, or hard to steer reliably.
intelligence-collection-disciplines
Run competitive research like an intelligence agency: eight collection disciplines (OSINT to MASINT), signal-to-inference chains, and fusion. Use when one-source research isn't enough.
lean-ux-canvas
Guide teams through Lean UX Canvas v2. Use when framing a business problem, surfacing assumptions, and defining what to learn next.