Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Calibre-Labs/reforge-ai-evals --skill trace-codesgit clone --depth 1 https://github.com/Calibre-Labs/reforge-ai-evalsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/trace-codes)<a href="https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/trace-codes"><img src="https://agentmods.dev/badge/skills/calibre-labs/reforge-ai-evals/trace-codes/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/calibre-labs/reforge-ai-evals/trace-codes"><img src="https://agentmods.dev/badge/skills/calibre-labs/reforge-ai-evals/trace-codes.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00117 | $0.01894 |
| Opus 5.5 | $0.00047 | $0.00758 |
| Sonnet 5.5 | $0.00023 | $0.00379 |
| Haiku 4.5 | $0.00012 | $0.00189 |
Grade A, and why
trace-codes scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 165 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Trace codes skill
Trace analysis has three steps: build a set of inputs and run the agent, read the traces and write notes, then cluster the notes into codes. A person does the reading. This skill does the clustering and everything that follows from it.
A trace code is a short name for one pattern seen across traces, defined so that anyone can answer yes or no for a single trace. Codes become the vocabulary for the rubric, the backlog and the automated evals.
Do not write the notes yourself. If the user has traces and no notes, tell them to read and annotate at least ten first. Reading is where product judgment comes in, so a person does it. You may help them read faster (summarize a long trace, point at the tool calls) but the observations must be theirs.
Inputs
| Input | Required | Typical form |
|---|---|---|
| Notes | Yes | A CSV or JSON with a trace id and a free-text note per trace, pasted text, or a review page that stores notes (see below). |
| Traces | Helpful | The files or folder the notes refer to, so examples can quote what happened |
| Rubric | Helpful | The current eval rubric, to map each code to a line |
| Existing codes | If any | A previous trace codes file to merge into |
Find these in the working folder before asking. If the notes have no trace ids, number them and say so.
Workflow
Step 1. Split notes into observations
One note often contains several observations ("stale revenue number, also no source for the third company, good scoping question"). Split each note into single observations and keep the trace id with each. Keep positive observations too. A code can describe something the agent should keep doing.
Step 2. Cluster into codes
Group observations that describe the same pattern. For each group write:
- Title: three to six words, descriptive, no jargon the team would not use.
- Definition: one sentence that can be answered yes or no for a single trace. Start with "The agent ..." or "The output ...".
- Examples: two or three, each with the trace id and a short paraphrase of what happened. Quote the trace when it is available.
- Count: how many traces show it, out of how many were annotated.
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 7d ago Changed · +25 lines b7c58caed82a
- 10d ago First seen · 140 lines · 117 tokens per session scan A a5a3d8e26ee6
trace-codes is a skill published in the GitHub repository Calibre-Labs/reforge-ai-evals (87 stars, last pushed yesterday), licensed MIT. It adds 117 tokens to every session and 1,894 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-10-17.
Other skills, from other repositories
ai-shaped-readiness-advisor
Assess whether your product work is AI-first or AI-shaped. Use when evaluating AI maturity and choosing the next team capability to build.
agent-orchestration-advisor
Design multi-agent AI workflows with clear boundaries, handoffs, and monitoring. Use when a complex PM task should run as parallel specialized agents instead of one linear process.
company-intel
Research a company, industry, or competitor set using web search and seven analytical lenses. Use when you need structured intel that feeds downstream PM skills.
context-engineering-advisor
Diagnose context stuffing vs. context engineering. Use when an AI workflow feels bloated, brittle, or hard to steer reliably.
business-health-diagnostic
Diagnose SaaS business health across growth, retention, efficiency, and capital. Use when preparing a business review or prioritizing urgent fixes.
derisk-measurement-advisor
Identify what to measure, test, or track to de-risk a product or AI idea. Use when stress-testing an idea across internal (DUFV) and external (PESTEL) dimensions.