eval-integrity

eval-integrity — Benchmark Credibility Audit

User-invokable auditor for LLM evaluation and benchmark repos. It answers one question: if you published this benchmark's numbers, would they survive an adversarial reviewer?

Most benchmark repos have a runner and a leaderboard but lack the integrity scaffolding that makes a published score mean what it appears to mean. This skill checks for that scaffolding, names what is missing, and rates whether each gap invalidates published numbers or is nice-to-have.

The checks below were extracted from hardening a real agent benchmark (cot-bench, a private podcast-evals repo) for an external grant review. Where a check needs a concrete example, cot-bench file names appear as illustrations — they are hints for what good looks like, not paths the auditor expects to find in your repo.

This is the static-analysis cousin of code-review: same parallel-subagent orchestration, but the subagents audit eval methodology instead of code correctness.

---

Invocation: deliberately model-invocable — read-only audit; it reports and offers fixes, never edits the benchmark.

When to Use

• Before submitting a benchmark to a grant, conference, or public leaderboard.
• When someone says "I don't trust those numbers" and you need to know if they're right.
• After building an eval, before publishing the first headline result.
• Periodically, as a benchmark accretes models and the leaderboard becomes load-bearing.

When NOT to Use

• A repo that is not an eval or benchmark (no scoring, no leaderboard, no judge). This skill has nothing to grade.
• A toy eval with no published numbers and no intent to publish. Integrity scaffolding is overhead until someone relies on the score.

---

Step 1: Confirm the target is a benchmark and locate its parts

Target: $ARGUMENTS (path to the benchmark repo; if empty, assume the current directory).

Establish the repo is gradable, then map where the audit will look. Run from the target repo root: