Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/hashgraph-online/awesome-codex-plugins/eval-harnessnpx skills add hashgraph-online/awesome-codex-plugins --skill eval-harnessgit clone --depth 1 https://github.com/hashgraph-online/awesome-codex-pluginsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/hashgraph-online/awesome-codex-plugins/eval-harness)<a href="https://agentmods.dev/skills/hashgraph-online/awesome-codex-plugins/eval-harness"><img src="https://agentmods.dev/badge/skills/hashgraph-online/awesome-codex-plugins/eval-harness.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00043 | $0.01288 |
| Opus 5 | $0.00022 | $0.00644 |
| Sonnet 5 | $0.00009 | $0.00258 |
| Haiku 4.5 | $0.00004 | $0.00129 |
Grade A, and why
eval-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured today.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 188 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Harness Skill
一个用于 Claude Code 会话的正式评估框架，实现 eval-driven development (EDD) 原则。
何时激活
- 为 AI 辅助工作流设置 EDD
- 定义 Claude Code 任务完成的 pass/fail 标准
- 使用 pass@k 指标测量 agent 可靠性
- 为 prompt 或 agent 更改创建回归测试套件
- 跨模型版本基准测试 agent 性能
理念
Eval-Driven Development 将评估视为"AI 开发的单元测试":
- 在实现前定义预期行为
- 在开发期间持续运行评估
- 用每次变更追踪回归
- 使用 pass@k 指标测量可靠性
评估类型
Capability Evals
测试 Claude 能否做以前不能做的事:
[CAPABILITY EVAL: points-calculation]
Task: 计算用户积分并确定等级
Success Criteria:
- [ ] 积分正确累加
- [ ] 等级边界正确
- [ ] 权益解锁逻辑正确
Expected Output: 用户总积分 = 1500，等级 = L3
Regression Evals
确保变更不破坏现有功能:
[REGRESSION EVAL: login-flow]
Baseline: sha-abc123
Tests:
- existing-login: PASS
- session-management: PASS
- logout-flow: PASS
Result: 3/3 passed (previously 3/3)
Grader 类型
1. Code-Based Grader
使用代码的确定性检查:
# 检查文件是否包含预期模式
grep -q "export function handlePoints" src/points.ts && echo "PASS" || echo "FAIL"
# 检查测试是否通过
npm test -- --testPathPattern="points" && echo "PASS" || echo "FAIL"
2. Model-Based Grader
使用 Claude 评估开放式输出:
[MODEL GRADER PROMPT]
评估以下代码变更：
1. 它是否解决了陈述的问题？
2. 结构是否良好？
3. 边界情况是否处理？
4. 错误处理是否适当？
Score: 1-5 (1=差, 5=优秀)
Reasoning: [解释]
3. Human Grader
标记为手动审查:
[HUMAN REVIEW REQUIRED]
Change: 描述变更内容
Reason: 为什么需要人工审查
Risk Level: LOW/MEDIUM/HIGH
指标
pass@k
"k 次尝试中至少一次成功"
- pass@1: 首次尝试成功率
- pass@3: 3 次内成功
- 典型目标: pass@3 > 90%
pass^k
"所有 k 次试验都成功"
- 更高可靠性标准
- 用于关键路径
评估工作流
1. 定义（编码前）
## EVAL DEFINITION: points-system
### Capability Evals
1. 可以计算用户积分
2. 可以确定用户等级
3. 可以解锁权益
### Regression Evals
1. 现有登录仍然有效
2. 会话管理未改变
3. 登出流程完整
### Success Metrics
- pass@3 > 90% for capability evals
- pass^3 = 100% for regression evals
2. 实现
编写代码通过定义的评估。
3. 评估
# 运行 capability evals
[Run each capability eval, record PASS/FAIL]
# 运行 regression evals
npm test -- --testPathPattern="existing"
# 生成报告
4. 报告
EVAL REPORT: points-system
==========================
Capability Evals:
calculate-points: PASS (pass@1)
determine-level: PASS (pass@2)
unlock-benefits: PASS (pass@1)
Overall: 3/3 passed
Regression Evals:
login-flow: PASS
session-mgmt: PASS
logout-flow: PASS
Overall: 3/3 passed
Metrics:
pass@1: 67% (2/3)
pass@3: 100% (3/3)
Status: READY FOR REVIEW
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- today First seen · 188 lines · 43 tokens per session scan A bdec481c9fef
eval-harness is a skill published in the GitHub repository hashgraph-online/awesome-codex-plugins (924 stars, last pushed today), licensed Apache-2.0. It adds 43 tokens to every session and 1,288 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-05.
Other skills, from other repositories
search
Search 2500+ curated ChatGPT and LLM open-source repositories. Use when the user asks to find tools, libraries, or repos related to ChatGPT, LLMs, RAG, agents, langchain, NLP, AI development, or any open-source AI tooling.
similar-resources
Given a Japanese NLP GitHub repo or Hugging Face model/dataset (URL / owner/repo / tool name), find repositories or models/datasets that do the same or related processing. Mines the bundled dataset for content-similar items, then expands via web research across both GitHub and Hugging Face, then merges and re-ranks.
sprr
Single PR reviewer for awesome-quant. Use when the user asks to review, validate, comment on, label, close, or merge one specific pull request that adds README.md entries. Triggers include "sprr", "review PR", "check PR", and "validate contribution".
bprr
Bulk PR reviewer for awesome-quant. Use when the user asks to review all open PRs, review unreviewed PRs, bulk review, or mentions "bprr". Reviews open PRs lacking the reviewed label and presents a summary before any merge/comment/label action.
reverse-engineering-android-malware-with-jadx
Reverse engineers malicious Android APK files using JADX decompiler to analyze Java/Kotlin source code, identify malicious functionality including data theft, C2 communication, privilege escalation, and overlay attacks. Examines manifest permissions, receivers, services, and native libraries. Activates for requests…
implementing-code-signing-for-artifacts
This skill covers implementing code signing for build artifacts to ensure integrity and authenticity throughout the software supply chain. It addresses signing binaries, packages, and containers using GPG, Sigstore, and platform-specific signing tools, establishing trust chains, and verifying signatures in deployment…