What is skill-auditor?

A testing skill that checks whether a third-party coding-agent skill does what its documentation claims. It breaks claims into individual items and tests them in a temporary workspace.

Use it to audit a skill from a GitHub link or local path. It produces a claim-by-claim report without modifying the skill being tested.

It helps reveal when a skill is overstating its abilities or behaving differently from its description. Findings are recorded with evidence and classified as confirmed, refuted, unverifiable, or blocked.

skill-auditor — Skill for Claude Code and Codex by tlzmw001

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add tlzmw001/naiyue-skills --skill skill-auditor

Clone the repo

git clone --depth 1 https://github.com/tlzmw001/naiyue-skills

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/tlzmw001/naiyue-skills/skill-auditor/github.svg)](https://agentmods.dev/skills/tlzmw001/naiyue-skills/skill-auditor)

Your own site

<a href="https://agentmods.dev/skills/tlzmw001/naiyue-skills/skill-auditor"><img src="https://agentmods.dev/badge/skills/tlzmw001/naiyue-skills/skill-auditor/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/tlzmw001/naiyue-skills/skill-auditor"><img src="https://agentmods.dev/badge/skills/tlzmw001/naiyue-skills/skill-auditor.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 187 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 3,278 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00187	$0.03278
Opus 5	$0.00093	$0.01639
Sonnet 5	$0.00037	$0.00656
Haiku 4.5	$0.00019	$0.00328

Measured 12d ago against content hash c80a45d09452, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-12, from the pricing page.

Security

Grade A, and why

skill-auditor scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.

The scan reads SKILL.md. This mod also ships 5 executable files (scripts/claims.py, scripts/cleanup.py, scripts/init_workspace.py, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/skill-auditor/SKILL.md · 139 lines

How it starts

The opening of the file, as written. The whole thing — 139 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Skill Auditor:skill 一致性审计(loop engineering 实践)

回答一个问题:这个 skill 宣称的能力,与它实际的表现是否一致? 输出一份逐条声称、证据可追溯的审计报告。

核心架构:确定性骨架 + 一处受控自迭代

本 skill 的可信度不依赖 AI 自觉,依赖结构:

claims.json 是单一事实源(脊柱):驱动循环、累积证据、投影报告,三个职责一份数据。结构契约见 references/claim-schema.md,开工前先读它。
AI 只做局部判断(拆声称、构场景、跑任务、判单条状态、写归因);转不转、停不停、状态怎么落、报告怎么出,全部由 scripts/ 下的确定性脚本执行。
唯一的自迭代发生在循环体内:claim 的灰区触发下一轮针对性场景,直到灰区清零或撞预算。除此之外任何环节都不迭代——不迭代打磨结论(会漂离证据),不迭代改进被测 skill(那是优化不是验证),不为消灭"无法验证"而反复凑(那是诚实结论)。

铁律(全流程有效,任何阶段不得违反)

任何人不得直接编辑 claims.json。 写入只经 scripts/claims.py;状态与判停只经 scripts/update_states.py;报告只经 scripts/render_report.py。你(主 Agent)也不例外。
一切验证产物只落在 <workspace>/sandbox/ 内,按轮分子目录。报告与 claims.json 在沙箱外。流程结束用 scripts/cleanup.py 收尾——它默认把关键中间产物整理进 review/ 复盘包(供效果展示与 review),只删冗余沙箱;会先校验报告已落地,顺序不可颠倒。用户事后可用 --purge-all 彻底清空(连 claims.json 与报告)。
终态 claim(confirmed/refuted/unverifiable/blocked)永不重测、永不改判。
不修复、不优化被测 skill。 发现的问题进证据,改进建议只能出现在报告交付后的对话里,绝不回流驱动重跑。
两个硬确认点(interactive 模式)必须真正停下等用户回复,见下文 ⛔ 标记。介绍完就继续跑 = 违规。

工作区

<workspace>/                  # 建议 /tmp/skill-audit-<name>/ 或用户指定
├── claims.json               # 事实源
├── agent-outputs/round-N/    # 各子Agent 的结构化产出(合并进事实源的中转)
├── report-<name>.md          # 最终交付物
├── sandbox/                  # target/ + deps/ + round-N/,清理时删除
└── review/                   # 清理后保留的复盘包(reasoning/ produced/ inputs/),供效果展示与 review

主流程

执行下列命令前，将 SKILL_DIR 设为当前 skill-auditor/SKILL.md 所在目录；不要假设当前工作目录就是 skill 目录。

Phase 0:初始化(确定性)

python3 "$SKILL_DIR/scripts/init_workspace.py" <workspace> --source <GitHub URL 或本地路径> \
    [--mode interactive|auto] [--max-rounds 3] [--max-rounds-per-claim 2]

用户给链接就 clone,给路径就 copy,脚本自动处理并列出文档清单。

Phase 1:claim 抽取 → ⛔ 确认点 1

派子 Agent 按 references/claim-extraction.md 抽取原子声称,产出写入 agent-outputs/claims-extracted.json,然后:

python3 "$SKILL_DIR/scripts/claims.py" <workspace> import-claims --file agent-outputs/claims-extracted.json
python3 "$SKILL_DIR/scripts/claims.py" <workspace> show

Read the full file on GitHub · 139 lines

Files

What ships with it

13 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

12d ago First seen · 139 lines · 187 tokens per session scan A c80a45d09452

Subscribe to this mod's changes

skill-auditor is a skill published in the GitHub repository tlzmw001/naiyue-skills (5 stars, last pushed 1mo ago), licensed MIT. It adds 187 tokens to every session and 3,278 once invoked, about $0.0009 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

skill-auditor