harbor-framework/harbor

Framework for evaluating and improving agents

About the project

Harbor is a framework for evaluating and improving AI agents and language models across reusable benchmarks and execution environments. Developers and researchers use it to run agent evaluations, create benchmarks, perform parallel experiments, and generate rollouts for reinforcement-learning optimization, with the catalogue offering skills, instructions, and an MCP integration for the framework.

This repository also configures its own agents. See what harbor tells them →

6.2kStars on the repository
12Mods indexed here, across every type
todayLast push, which is what freshness is scored on
Apache-2.0Licence, which decides whether bodies are shown

bundled-keep

01

harbor-framework/harbor

Skill Claude CodeCodex

Existing task skill that should remain after job-level skill injection.

not rated 6.2k +197 today A SkillSpector: pass 17 tokens original Apache-2.0

runtime-proof

02

harbor-framework/harbor

Skill Claude CodeCodex

Write the proof file for the Harbor runtime skill injection example.

not rated 6.2k +197 today A SkillSpector: pass 15 tokens original Apache-2.0

generate-greeting

03

harbor-framework/harbor

Skill Claude CodeCodex

Generate a greeting message and write it to a file.

not rated 6.2k +197 today A SkillSpector: pass 15 tokens original Apache-2.0

create-adapter

04

harbor-framework/harbor

Skill Claude CodeCodex

Scaffold a new Harbor benchmark adapter by running harbor adapter init and then guide implementation using the Adapters Agent Guide as the authoritative spec.

not rated 6.2k +197 changed 24d ago A SkillSpector: pass 34 tokens original Apache-2.0

create-task

05

harbor-framework/harbor

Skill Claude CodeCodex

Create a new Harbor task for evaluating agents. Use when the user wants to scaffold, build, or design a new task, benchmark problem, or eval. Guides through instruction writing, environment setup, verifier design (pytest vs Reward Kit vs custom), and solution scripting.

not rated 6.2k +197 changed 9d ago F ✓ AI review Socket: passSnyk: passSkillSpector: warn 56 tokens original Apache-2.0

harbor-exec

06

harbor-framework/harbor

Skill Claude CodeCodex

Use when working with Harbor's harbor exec CLI workflow: compiling files, directories, or globs into Harbor tasks; running map jobs; configuring artifacts and existence-only verification; using map-reduce; writing or reviewing ExecConfig YAML/JSON/TOML; or debugging command behavior, config validation, and job outputs.

not rated 6.2k +197 today A SkillSpector: pass 71 tokens original Apache-2.0

publish

07

harbor-framework/harbor

Skill Claude CodeCodex

Publish a Harbor task or dataset to the registry. Use when the user wants to upload, publish, or share tasks or datasets/benchmarks on the Harbor registry.

not rated 6.2k +197 changed 1mo ago A SkillSpector: warn 35 tokens original Apache-2.0

rewardkit

08

harbor-framework/harbor

Skill Claude CodeCodex

Write Harbor task verifiers using Reward Kit. Use when creating or editing a task's tests/ directory, adding grading criteria, setting up LLM/agent judges, or designing verifiers that produce a reward score.

not rated 6.2k +197 changed 7d ago A Socket: passSnyk: passSkillSpector: pass 46 tokens original Apache-2.0

harbor-framework/harbor

Skill Claude CodeCodex

Create or reuse Hugging Face dataset PRs for harborframework/parity-experiments and upload Harbor parity/oracle result folders efficiently with sparse checkout, raw git pushes, and Git LFS.

not rated 6.2k +197 today A SkillSpector: pass 49 tokens original Apache-2.0

At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: