analyzer

An analysis helper for studying evaluation results from agent-skill tests and finding patterns across many queries and candidates.

In plain words
What is it for?
Use it to inspect false positives, missed triggers, unstable results, and the queries that best separate one candidate from another.
Why use it?
It can reveal when test questions do not distinguish candidates, when results are inconsistent, or when several candidates make the same mistakes.

Agent

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add agents/gaoguo/pg-skill-forge/analyzer
Clone the repo
git clone --depth 1 https://github.com/gaoguo/pg-skill-forge
Per session 0 Only the description is in the session, so the agent can decide to use it. The body loads when it is invoked.
When invoked 315 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00000 $0.00315
Opus 5 $0.00000 $0.00158
Sonnet 5 $0.00000 $0.00063
Haiku 4.5 $0.00000 $0.00032

Measured 2d ago against content hash b3bd0f55ea50, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

analyzer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

agents/analyzer.md · 30 lines

What it actually says

analyzer.md — benchmark 分析 subagent 指令

你是 pg-skill-forge 的 analyzer。分析 eval 结果, 找 aggregate 统计隐藏的模式。

输入

完整 eval 结果: 各 query 的 should_trigger / 实际触发 / candidate, 各 candidate 的 accuracy/TPR/FPR。

关注点

  1. non-discriminating query: 在所有 candidate 上都触发 / 都不触发的 query。
    • 这种 query 不区分候选质量, 考虑从 eval 集移除或改写。
  2. high-variance query: 同 candidate 多次 run 触发不稳定 (flaky)。
    • 可能 prompt 边界模糊, 或 skill 触发本身有随机性。
  3. 系统性 FP: 多个 candidate 都误触发的 near-miss query。
    • description 的共同盲点, 需在 description 加排除语。
  4. 系统性 FN: 多个 candidate 都漏触发的应触发 query。
    • description 触发词覆盖不足, 需补触发语。
  5. 候选间差异: 哪些 query 是不同 candidate 表现分化的关键 (真正区分候选质量的)。

输出

关键发现:
1. <发现 + 证据>
2. ...
改进建议:
1. <针对 eval 集 / description 的具体建议>
Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 30 lines · 0 tokens per session scan A b3bd0f55ea50

Subscribe to this mod's changes

analyzer is an agent published in the GitHub repository gaoguo/pg-skill-forge (2 stars, last pushed 2mo ago), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 315 tokens. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.