Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/jmagly/aiwg/eval-reportnpx skills add jmagly/aiwg --skill eval-reportgit clone --depth 1 https://github.com/jmagly/aiwgWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/jmagly/aiwg/eval-report)<a href="https://agentmods.dev/skills/jmagly/aiwg/eval-report"><img src="https://agentmods.dev/badge/skills/jmagly/aiwg/eval-report.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00021 | $0.01150 |
| Opus 5 | $0.00010 | $0.00575 |
| Sonnet 5 | $0.00004 | $0.00230 |
| Haiku 4.5 | $0.00002 | $0.00115 |
Grade A, and why
eval-report scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 159 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Evaluation Report
Generate a quality report from accumulated evaluation results.
Research Foundation
- REF-001: BP-9 - Continuous evaluation of agent performance
- REF-002: KAMI benchmark methodology for real agentic task evaluation
Usage
/eval-report
/eval-report --output .aiwg/reports/quality-report.md
/eval-report --compare previous-report.json
/eval-report --mode sdlc --format json
Options
| Option | Default | Description |
|---|---|---|
| --output | stdout | Output file path |
| --compare | none | Previous report to diff against |
| --mode | all | Agent category: sdlc, marketing, forensics, all |
| --format | markdown | Output format: markdown, json |
| --since | none | Only include results after this date (ISO 8601) |
| --threshold | 0.85 | Score below this triggers a warning |
Process
- Collect Results: Read all
eval-*.jsonfiles from.aiwg/reports/ - Aggregate Scores: Compute per-agent and per-archetype scores
- Detect Regressions: Compare against --compare baseline if provided
- Rank Agents: Sort by overall score, flag below-threshold agents
- Build Recommendations: Surface specific agents and archetypes needing attention
- Output Report: Write markdown or JSON to --output or stdout
Report Sections
Summary Dashboard
Overall health at a glance — total agents tested, aggregate score, regression count.
By Archetype
Pass rates per Roig (2025) failure archetype across all agents.
Agents Needing Attention
Agents below the --threshold, with consecutive-failure streaks flagged.
Regression Analysis
When --compare is provided: agents whose scores dropped since the baseline.
Recommendations
Prioritized action list: which agents to review, which archetypes to harden.
Output Format (Markdown)
# Agent Quality Report
**Generated**: 2026-04-01T10:30:00Z
**Agents Tested**: 58
**Overall Score**: 87%
**Regressions**: 2
## By Archetype
| Archetype | Pass Rate | Trend |
|-----------|-----------|-------|
| #1 Grounding | 92% | ↑ |
| #2 Substitution | 88% | → |
| #3 Distractor | 78% | ↓ |
| #4 Recovery | 90% | ↑ |
## Agents Needing Attention
| Agent | Score | Consecutive Failures | Issue |
|-------|-------|---------------------|-------|
| data-analyst | 72% | 3 | distractor-test |
| api-designer | 79% | 1 | latency regression (+40%) |
## Recommendations
1. Review `data-analyst` context filtering — failed distractor-test 3 consecutive runs
2. Investigate `api-designer` tool selection — latency regression
3. Increase distractor-test scenarios for marketing agents (78% pass rate below 80% target)
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 159 lines · 21 tokens per session scan A 04a45998b17a
eval-report is a skill published in the GitHub repository jmagly/aiwg (209 stars, last pushed yesterday), licensed MIT. It adds 21 tokens to every session and 1,150 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
vertical-real-estate
Residential-proptech domain knowledge so architect / pm aren't naive when speccing real-estate products (listings, lead-crm, transaction-coordination, property-mgmt). Codifies MLS/IDX reality, listing status lifecycle + syndication canonical-source, long-cycle lead nurture, transaction-coordination as the high-pain…
skeptical-triage
Reusable 3-round self-challenge + arbiter pattern for filtering false positives from findings/verdicts. Use when the cost of a false-positive gate block exceeds the cost of 4 extra LLM turns.
vertical-hr-recruiting
Domain-knowledge primer for the HR & recruiting vertical (ATS, onboarding, workforce scheduling, engagement). Applied by architect/pm during spec authoring so they aren't naive about hiring pipelines, the admitted offer→onboard data-carry gap, EEO/I-9 compliance, and shift-coverage rules. Stops the four products from…
lifecycle-messaging
Email/SMS lifecycle and deliverability framework for SMB Product-Builder products that send transactional or lifecycle messages (booking reminders, CRM sequences, receipts, win-back). Codifies provider selection (Resend/Postmark/Twilio/SendGrid), domain auth (SPF/DKIM/DMARC), consent and compliance (TCPA, CAN-SPAM…
pm-planning
Decomposition methodology for pm agent — turns an approved ARCH document into a Beads task list with explicit dependencies, time-boxes, and acceptance criteria. The pipeline can only orchestrate work it can see; this skill defines what "seeable work" looks like.
well-architected
6-pillar architecture review framework. Adapted from AWS Well-Architected for use by greatcto's architect agent on every non-nano ARCH document. Forces explicit answers across operational excellence, security, reliability, performance, cost, and sustainability — not just feature design.