harbor-adapter-creator

Creating Harbor Benchmark Adapters

Adapters convert external benchmarks (SimpleQA, GAIA, AiderPolyglot, CodePDE, spider2-dbt, etc.) into Harbor's standardized task directory format. Each adapter reads source benchmark data and generates many individual task directories, one per benchmark instance.

When to Use an Adapter vs. Creating Tasks Directly

Use an adapter when:

• You have an existing benchmark dataset with many instances
• Tasks share the same structure but differ in data (questions, code, etc.)
• You want to track evaluation parity with the original benchmark

Create tasks directly when:

• You're authoring original evaluation challenges
• Each task has unique structure and environment
• There are fewer than ~10 tasks

Adapter Directory Structure

adapters/<adapter-id>/
├── adapter.py              # Core conversion logic (adapter class)
├── run_adapter.py          # CLI entry point
├── run_<adapter-id>.yaml   # Job config for oracle and parity experiment runs
├── README.md               # Benchmark docs, license, parity, citation
├── parity_experiment.json  # Parity tracking results (JSON array)
├── adapter_metadata.json   # Adapter metadata (JSON array)
└── template/               # Task template files
    ├── task.toml
    ├── instruction.md
    ├── environment/
    │   └── Dockerfile
    ├── tests/
    │   └── test.sh
    └── solution/
        └── solve.sh

All template files, parity_experiment.json, adapter_metadata.json, and README.md are required. The validator (harbor adapters validate) checks for all of them. The YAML job config is not validated but is expected for parity experiments.

Scaffolding with harbor adapters init

harbor adapters init my-benchmark

The interactive AdapterWizard prompts for:

• Benchmark name -- Vanilla name (e.g., "SWE-bench")
• Adapter ID -- Lowercase, no spaces (e.g., swebench)
• Class name -- Python class name (auto-derived or overridden, e.g., SwebenchAdapter)
• Description -- One-line description for the README
• Source URL -- Link to original benchmark paper or repo
• License -- Dataset license for the README