ai-scientist-evaluator

AI Scientist Evaluator

Use this skill when Codex should behave like a skeptical reviewer panel rather than a research generator. Evaluate completed outputs, not just plans.

Instructions

1. Confirm the request is evaluative. Use this skill to audit or compare existing outputs, not to perform the original research task.
2. Restate the exact task in one or two sentences so the review stays anchored to the real objective and required deliverables.
3. Inventory the submitted artifacts and note what is missing. Prefer primary artifacts over summaries:
   • notebooks, code, scripts, and workflow files
   • environment files, package versions, and runtime logs
   • figures, tables, and manuscript drafts
   • data provenance, accession lists, database versions, and citations
   • benchmark results, hardware notes, and task constraints
4. Choose the closest task profile from references/task_profiles.md and load the matching weights from assets/default_weight_profiles.yaml. Use the primary scientific profile first for composite tasks, then add manuscript comments as a secondary layer.
5. Review with a four-person panel and synthesize a consensus:
   • scientific validity reviewer
   • computational and reproducibility reviewer
   • domain biology reviewer
   • writing and editorial reviewer
6. Apply hard gates before generous scoring. A submission is not publication-ready if required deliverables are missing, claims are not supported by visible outputs, provenance is untraceable, the core method is not rerunnable, or the submission solves an easier adjacent problem.
7. Interrogate the submission with the relevant sections of references/question_bank.md. Always include the universal questions, then add the profile-specific and multi-submission questions when needed.
8. Scan for integrity, rigor, and validity problems using references/red_flags.md. Penalize missing evidence, task drift, unsupported biological claims, fabricated identifiers, and unverifiable citations more than polished narrative.
9. Score each category on the anchored 0 to 5 scale in references/score_scale.md. Use references/category_definitions.md if category meaning is unclear. A score of 5 earns the full category weight.
10. Convert the category scores to a weighted total out of 100. Do not trust submitted weight or weighted_points values. The bundled aggregator validates the review against evaluation_schema.json, loads the selected weight profile, and recomputes each weighted value. It accepts overall.total_score_100 only as a downward penalty. It ignores any score above the recomputed total. Apply penalties sparingly and explain them when category scores do not already capture the issue.
11. For multiple submissions, score each one independently before ranking. Use tie-breaks in this order:
    • fewer integrity or reproducibility problems
    • better satisfaction of the task's main objective
    • stronger validation or benchmarking
    • clearer limitation handling
    • better writing only after science and evidence are settled
12. Produce a concise consensus verdict with actionable revisions. Ground the review in concrete evidence from files, notebook cells, figure numbers, accessions, parameters, and versioned tools whenever possible.
13. When a structured artifact is useful, start from assets/evaluation_template.json and validate the shape against assets/evaluation_schema.json. Use assets/report_template.md for markdown reports. For completed JSON reviews, you may aggregate rankings with uv run --script "$HOME/.agents/skills/ai-scientist-evaluator/scripts/aggregate_reviews.py" review1.json review2.json --out_md leaderboard.md.