Awesome Codex Plugins is a curated catalogue and marketplace of installable plugins, skills, and related resources for OpenAI Codex and ChatGPT. It helps people find, create, validate, and submit extensions for Codex workflows across its app, command-line interface, and IDE extensions. The catalogue add-ons are the plugins, skills, agents, and hooks listed or supported by this project.
Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add hashgraph-online/awesome-codex-plugins --skill eval-harnessgit clone --depth 1 https://github.com/hashgraph-online/awesome-codex-pluginsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/hashgraph-online/awesome-codex-plugins/eval-harness)<a href="https://agentmods.dev/skills/hashgraph-online/awesome-codex-plugins/eval-harness"><img src="https://agentmods.dev/badge/skills/hashgraph-online/awesome-codex-plugins/eval-harness/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/hashgraph-online/awesome-codex-plugins/eval-harness"><img src="https://agentmods.dev/badge/skills/hashgraph-online/awesome-codex-plugins/eval-harness.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00043 | $0.01288 |
| Opus 5.5 | $0.00017 | $0.00515 |
| Sonnet 5.5 | $0.00009 | $0.00258 |
| Haiku 4.5 | $0.00004 | $0.00129 |
Grade A, and why
eval-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 188 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Harness Skill
一个用于 Claude Code 会话的正式评估框架，实现 eval-driven development (EDD) 原则。
何时激活
- 为 AI 辅助工作流设置 EDD
- 定义 Claude Code 任务完成的 pass/fail 标准
- 使用 pass@k 指标测量 agent 可靠性
- 为 prompt 或 agent 更改创建回归测试套件
- 跨模型版本基准测试 agent 性能
理念
Eval-Driven Development 将评估视为"AI 开发的单元测试":
- 在实现前定义预期行为
- 在开发期间持续运行评估
- 用每次变更追踪回归
- 使用 pass@k 指标测量可靠性
评估类型
Capability Evals
测试 Claude 能否做以前不能做的事:
[CAPABILITY EVAL: points-calculation]
Task: 计算用户积分并确定等级
Success Criteria:
- [ ] 积分正确累加
- [ ] 等级边界正确
- [ ] 权益解锁逻辑正确
Expected Output: 用户总积分 = 1500，等级 = L3
Regression Evals
确保变更不破坏现有功能:
[REGRESSION EVAL: login-flow]
Baseline: sha-abc123
Tests:
- existing-login: PASS
- session-management: PASS
- logout-flow: PASS
Result: 3/3 passed (previously 3/3)
Grader 类型
1. Code-Based Grader
使用代码的确定性检查:
# 检查文件是否包含预期模式
grep -q "export function handlePoints" src/points.ts && echo "PASS" || echo "FAIL"
# 检查测试是否通过
npm test -- --testPathPattern="points" && echo "PASS" || echo "FAIL"
2. Model-Based Grader
使用 Claude 评估开放式输出:
[MODEL GRADER PROMPT]
评估以下代码变更：
1. 它是否解决了陈述的问题？
2. 结构是否良好？
3. 边界情况是否处理？
4. 错误处理是否适当？
Score: 1-5 (1=差, 5=优秀)
Reasoning: [解释]
3. Human Grader
标记为手动审查:
[HUMAN REVIEW REQUIRED]
Change: 描述变更内容
Reason: 为什么需要人工审查
Risk Level: LOW/MEDIUM/HIGH
指标
pass@k
"k 次尝试中至少一次成功"
- pass@1: 首次尝试成功率
- pass@3: 3 次内成功
- 典型目标: pass@3 > 90%
pass^k
"所有 k 次试验都成功"
- 更高可靠性标准
- 用于关键路径
评估工作流
1. 定义（编码前）
## EVAL DEFINITION: points-system
### Capability Evals
1. 可以计算用户积分
2. 可以确定用户等级
3. 可以解锁权益
### Regression Evals
1. 现有登录仍然有效
2. 会话管理未改变
3. 登出流程完整
### Success Metrics
- pass@3 > 90% for capability evals
- pass^3 = 100% for regression evals
2. 实现
编写代码通过定义的评估。
3. 评估
# 运行 capability evals
[Run each capability eval, record PASS/FAIL]
# 运行 regression evals
npm test -- --testPathPattern="existing"
# 生成报告
4. 报告
EVAL REPORT: points-system
==========================
Capability Evals:
calculate-points: PASS (pass@1)
determine-level: PASS (pass@2)
unlock-benefits: PASS (pass@1)
Overall: 3/3 passed
Regression Evals:
login-flow: PASS
session-mgmt: PASS
logout-flow: PASS
Overall: 3/3 passed
Metrics:
pass@1: 67% (2/3)
pass@3: 100% (3/3)
Status: READY FOR REVIEW
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 188 lines · 43 tokens per session scan A bdec481c9fef
eval-harness is a skill published in the GitHub repository hashgraph-online/awesome-codex-plugins (1,397 stars, last pushed today), licensed Apache-2.0. It adds 43 tokens to every session and 1,288 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-05.
Other skills, from other repositories
awesome-chatgpt
Search this repository's awesome-ChatGPT-repositories list for open-source ChatGPT and LLM projects such as RAG frameworks, agents, chatbots, CLIs, prompts, and browser extensions. Use when the user asks to find tools, libraries, or repos related to ChatGPT, LLMs, or AI development; not for editing or adding list…
awesome-chatgpt-search
Search 2500+ curated ChatGPT and LLM open-source repositories. Use when the user asks to find tools, libraries, or repos related to ChatGPT, LLMs, RAG, agents, langchain, NLP, AI development, or any open-source AI tooling.
tdd-test-engineer
Use for test-first development, regression tests, flaky test debugging, coverage gaps, test strategy, CI failures, or converting bugs into minimal reproducible tests.
add-llm-evals
Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI)…
build-eval-dataset
Use this to build a good evaluation dataset for an LLM app, the part everyone underestimates. Trigger on "make an eval set", "what should I test my LLM on", "I don't have test data for my prompt", "build a golden dataset", or before setting up evals. A great eval set beats a great metric; garbage-in means your evals…
set-up-ab-testing
Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Measure impact on real…