benchmark-reference

Handle Benchmark Reference

Reference files live at .claude/references/datasets_and_tasks/{name}.md. The index lives at .claude/references/datasets_and_tasks/datasets_and_tasks_map.md.

The argument has two parts:

1. Benchmark name (required) — first word or quoted phrase (e.g., Countdown, "GSM8K", "SWE-bench Verified")
2. Context (optional) — everything after the name. This is either:
   • A correction ("Countdown evaluation should check equation validity, not direct string match")
   • Additional info to merge ("add that GSM8K has 8.5K training examples")
   • Empty — means "create from scratch" or "review and enrich existing"

Step 1: Determine mode

IF reference file exists for this benchmark:
  IF context provided → UPDATE mode (apply correction / add info)
  IF no context      → ENRICH mode (research and fill gaps)
ELSE:
  CREATE mode (build from scratch)

Read the existing file if it exists. Read datasets_and_tasks_map.md to check for an entry.

Normalize the benchmark name to a filename: lowercase, hyphens for spaces, no special chars. Example: "SWE-bench Verified" → swe-bench-verified.md

Step 2: Research (CREATE and ENRICH modes)

Use web search and paper search to find authoritative sources. Prioritize:

1. The original paper introducing the benchmark
2. Official HuggingFace dataset page (if it exists)
3. Official code repository
4. Leaderboard or evaluation harness docs (e.g., lm-evaluation-harness, BigCode, HELM)

Gather enough to fill every section in the template below. If a section cannot be filled, mark it *Unknown — needs investigation*.

Step 3: Write or update the reference file

CREATE mode

Write a new file using this template:

# {Benchmark Name}

## Overview

{2-3 sentence description: what does this benchmark measure, who introduced it, why does it matter}

- **Introduced in:** {Paper title} ({Authors}, {Year}) — [{arxiv link}]({url})
- **Official HuggingFace:** [{org/dataset}]({url}) or *None*
- **Code repo:** [{org/repo}]({url}) or *None*
- **Leaderboard:** [{name}]({url}) or *None*

## Task Description

{What does the model need to do? Be specific. Include input/output format, number of examples/splits, any subtasks or difficulty levels.}

## Evaluation

### Method
{How are answers scored? Direct string match, equation equivalence, LLM-as-judge, execution-based, F1, exact match, pass@k, etc. Be precise — this is what experiment code will rely on.}

### Metric(s)
{Primary metric and any secondary metrics. Include formulas or references if non-obvious.}

### Known Pitfalls
{Common mistakes in evaluation: wrong normalization, off-by-one in pass@k, answer extraction regex failures, etc.}

## Running It

### Requirements
{Does it need Docker, a sandbox, network access, a compiler, specific packages? Is it agentic / multi-turn? How long does a typical run take?}

### Recommended Setup
{Best way to run: lm-eval-harness task name, custom script, official repo instructions, etc. Link to working configs if available.}

### Prompt / Few-Shot Format
{Reference prompt template or describe the standard format. Include few-shot count if conventional. If there's a canonical prompt, quote it in a code block.}

## Typical Results

{Brief table or notes on SOTA / baseline performance ranges so we know if our numbers are in the right ballpark.}

| Model | Score | Source |
|-------|-------|--------|

## Setup Checklists

Context-specific questions to resolve BEFORE writing code. These prevent common mistakes and wasted compute by surfacing ambiguities early. Not every benchmark needs all three sections — include only those that apply.

### For Evaluation
{Bulleted list of questions you should answer before writing an evaluation script for this task. Focus on: answer extraction, scoring edge cases, metric configuration, dataset split selection, prompt format.}

### For Distillation / Data Generation
{Bulleted list of questions you should answer before generating training data from this task. Focus on: output format, trace style, filtering criteria, dataset size, deduplication, quality checks.}

### For RL Training
{Bulleted list of questions you should answer before setting up an RL training run with this task as the reward signal. Focus on: reward function design, format reward, partial credit, hyperparameter choices tied to the task, known failure modes.}

## Notes

{Anything else: known dataset contamination issues, version differences, community conventions, licensing.}