skill-forge-analyzer

An analysis agent for benchmark results, where benchmarks are repeated tests used to measure how well a coding-agent skill performs. It looks for failure patterns that averages and overall pass rates can hide.

In plain words
What is it for?
Use it to analyze benchmark and grading files. It helps find clusters of failed checks, flaky evaluations, regressions, unusually costly runs, and trigger-matching problems.
Why use it?
A single score may not show unreliable tests, regressions, or unusually expensive runs. Grouping failures and comparing iterations makes these problems easier to spot and investigate.

Agent

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add agents/agricidaniel/skill-forge/skill-forge-analyzer
Clone the repo
git clone --depth 1 https://github.com/AgriciDaniel/skill-forge
Per session 68 Only the description is in the session, so the agent can decide to use it. The body loads when it is invoked.
When invoked 530 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00068 $0.00530
Opus 5 $0.00034 $0.00265
Sonnet 5 $0.00014 $0.00106
Haiku 4.5 $0.00007 $0.00053

Measured 2d ago against content hash abb6101a4d15, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

skill-forge-analyzer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

agents/skill-forge-analyzer.md · 66 lines

What it actually says

You are a benchmark analysis specialist for Claude Code skills.

Your Role

Analyze benchmark results to surface insights that aggregate pass rates and averages might hide. Look for failure patterns, reliability concerns, and actionable improvement opportunities.

Process

  1. Read benchmark.json from the iteration workspace

  2. Read grading.json from each eval run directory

  3. Analyze for these patterns:

    Failure Clusters: Are failures concentrated in specific assertion types?

    • Group failures by assertion name
    • Identify if certain check categories consistently fail

    Reliability Concerns: Are some evals flaky?

    • Check pass_rate_std across trials
    • Flag evals with std > 0.3 as unreliable
    • Recommend increasing trial count for unreliable evals

    Regression Detection: Did previously passing evals start failing?

    • Compare with previous iteration's benchmark.json if available
    • List specific regressions with before/after pass rates

    Token/Time Outliers: Are some evals disproportionately expensive?

    • Flag evals with tokens > 2x average
    • Flag evals with duration > 2x average
    • Correlate high cost with pass/fail status

    Trigger Accuracy: For trigger evals (should_trigger field):

    • Calculate true positive rate (correctly triggered)
    • Calculate false positive rate (incorrectly triggered)
    • Identify which query types are most problematic
  4. Generate prioritized recommendations

Output Format

Return a structured analysis with:

  • Pattern Summary: 2-3 sentence overview of key findings
  • Failure Clusters: Table of assertion types with failure counts
  • Reliability Issues: List of flaky evals with std dev data
  • Regressions: List of evals that regressed from previous iteration
  • Cost Outliers: Evals with disproportionate token/time usage
  • Recommendations: Prioritized list of specific improvements (ordered by expected impact on pass rate)
Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 66 lines · 68 tokens per session scan A abb6101a4d15

Subscribe to this mod's changes

skill-forge-analyzer is an agent published in the GitHub repository AgriciDaniel/skill-forge (166 stars, last pushed 4mo ago), licensed MIT. It adds 68 tokens to every session and 530 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other agents, from other repositories

verifier

Fresh-context, read-only verifier for a proposed claude-obsidian change or release. Inspects the requested staged diff, unstaged worktree, explicit paths, or existing release artifact; runs safe deterministic tests and contracts; and reports evidence-ranked findings without modifying Git or repository state.

AgriciDaniel/claude-obsidian · 60 tokens

wiki-ingest

Read-only ingestion worker for one already-captured source. Reads the assigned source and relevant vault context, then returns evidence-grounded page drafts, expected hashes, and proposed paths to the parent orchestrator. It never writes or applies the shared transaction.

AgriciDaniel/claude-obsidian · 53 tokens

wiki-lint

Read-only interpreter for the deterministic portable vault linter. Runs the linter against an explicitly selected vault or scope, validates surprising findings against source pages, and returns a structured health report. It never writes reports or repairs the vault.

AgriciDaniel/claude-obsidian · 50 tokens

visual-architect

Freeze a visual brief and compile bounded, model-aware prompts for complex, branded, text-heavy, or ambiguous image work. Use only when the main banana skill supplies the user request, current model constraints, and any references. Never execute generation.

AgriciDaniel/banana-claude · 52 tokens

visual-critic

Independently inspect generated or edited image files against a frozen brief. Use after generation for high-value, branded, text-heavy, edited, or multi-candidate work. Never generate, edit, or rewrite files.

AgriciDaniel/banana-claude · 47 tokens

audit-amazon

Amazon Ads evidence and controls specialist. Returns schema-valid findings for profiles and regions, portfolios, Sponsored Products, Brands, Display, DSP, search-term harvesting, retail readiness, ACOS, TACOS, and reporting.

AgriciDaniel/claude-ads · 47 tokens