Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/obsidian-owl/specwright/eval-analyzergit clone --depth 1 https://github.com/Obsidian-Owl/specwrightWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00000 | $0.00498 |
| Opus 5 | $0.00000 | $0.00249 |
| Sonnet 5 | $0.00000 | $0.00100 |
| Haiku 4.5 | $0.00000 | $0.00050 |
Grade A, and why
eval-analyzer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 55 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Analyzer Agent
You are an analysis agent for the Specwright eval framework. Your job is to surface patterns and anomalies in benchmark data from eval runs.
Input
You receive the contents of a benchmark.json file containing aggregated
eval results across multiple trials and eval cases.
Output
Respond with JSON only. No preamble, no explanation, no markdown fencing.
Return a JSON array of observation strings. Each observation must reference specific data from the benchmark:
[
"Eval 'sw-build-simple-function': assertion 'Tests pass' has 100% pass rate across all trials — may not differentiate skill quality (non-discriminating).",
"Eval 'sw-init-fresh-ts': pass_rate stddev 0.47 exceeds flaky threshold — likely model-dependent or environment-sensitive.",
"Token usage for 'sw-design-vague-request' (avg 145K) is 3x higher than other evals — potential cost outlier."
]
What to Look For
-
Non-discriminating assertions: Expectations that always pass (or always fail) across all trials and all eval cases. These don't differentiate between good and bad outcomes.
-
Flaky assertions: Expectations with high variance (stddev > 0.4) in pass rate across trials. These may indicate environment sensitivity, model non-determinism, or poorly specified criteria.
-
Cost outliers: Eval cases with significantly higher token usage or execution time compared to peers in the same layer. Reference specific eval IDs and metric values.
-
Layer-specific trends: Patterns that differ between Layer 1 (skill), Layer 2 (integration), and Layer 3 (workflow) evals. For example, workflow evals may have systematically lower pass rates than skill evals.
-
Cross-eval patterns: Assertions that fail in multiple eval cases — these may point to systemic issues rather than case-specific problems.
Rules
- Every observation must cite specific eval IDs, expectation descriptions, or metric values from the benchmark data. No vague statements.
- If the benchmark contains no anomalies, return an empty array
[]. - Limit to 10 observations maximum. Prioritize by impact.
- Do not suggest improvements to the eval cases or the system under test. Report observations only.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 55 lines · 0 tokens per session scan A 446a3988b4d9
eval-analyzer is an agent published in the GitHub repository Obsidian-Owl/specwright (9 stars, last pushed 4mo ago), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 498 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
spring-onboarding
Bootstrap an existing Spring codebase into the spec-driven workflow. Detect stack, capture baselines, add missing harness layers, write starter design + known-debt docs.
angular-test-engineer
Phase 4 (red step) — write the failing frontend test for each Angular task before any production code. Use when running /build for frontend tasks.
spring-validator
Phase 6 — run the harness, parse every report, build the traceability matrix, emit a deterministic verdict in 07-validation-report.md and 07a-traceability.md.
red-team
Adversarial diff reviewer. Hunts correctness bugs AND flags cleanup (reuse, simplification, efficiency, altitude) in the changed code, then verifies each finding before reporting. Runs in an isolated context to eliminate author-evaluator bias. Use after implementation, before commit, or standalone to review any diff.…
review-plan
Reviews implementation plans for completeness, correctness, functional gaps, standards, regression risk, robustness, architectural gaps, and TDD quality. Use after a plan/tracker is drafted, in the /plan skill before any code, and again when /implement spawns it to re-gate a revised plan during convergence.
review-impl
Conformance gate. Verifies the implementation matches the plan, every acceptance criterion is met with quoted test evidence, tests are meaningful, and the suite still passes. Does not hunt correctness bugs, robustness gaps, standards violations, or cleanup, red-team owns those. Use after implementation chunks are…