Harbor is a framework for evaluating and improving AI agents and language models across reusable benchmarks and execution environments. Developers and researchers use it to run agent evaluations, create benchmarks, perform parallel experiments, and generate rollouts for reinforcement-learning optimization, with the catalogue offering skills, instructions, and an MCP integration for the framework.
Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add harbor-framework/harbor --skill create-taskgit clone --depth 1 https://github.com/harbor-framework/harborWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/harbor-framework/harbor/create-task)<a href="https://agentmods.dev/skills/harbor-framework/harbor/create-task"><img src="https://agentmods.dev/badge/skills/harbor-framework/harbor/create-task/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/harbor-framework/harbor/create-task"><img src="https://agentmods.dev/badge/skills/harbor-framework/harbor/create-task.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- Socket pass
- Snyk pass
- NVIDIA SkillSpector warn
SkillSpector: 9 findings, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Privilege Escalation · line 54 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
- high Privilege Escalation · line 188 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
- high Tool Misuse · line 142 Tool calls are chained to bypass individual safety checks or escalate capabilities beyond what any single tool call would allow.Fix: Limit tool chaining depth and validate the output of each tool before passing it to the next. Require explicit user approval for multi-step chains.
- high Privilege Escalation · line 159 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
- medium MCP Rug Pull · line 128 uvx/uv tool run commands without ==version create a rug-pull risk.Fix: Pin the version: uvx package-name==1.2.3
- low Tool Misuse · line 69 Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).Fix: Validate all tool parameters against an allowlist. Reject dangerous parameter values (shell=True, --force, -rf /) and use safe defaults.
- low Tool Misuse · line 69 Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).Fix: Validate all tool parameters against an allowlist. Reject dangerous parameter values (shell=True, --force, -rf /) and use safe defaults.
- low Tool Misuse · line 69 Tool calls are chained to bypass individual safety checks or escalate capabilities beyond what any single tool call would allow.Fix: Limit tool chaining depth and validate the output of each tool before passing it to the next. Require explicit user approval for multi-step chains.
- low Supply Chain · line 141 Remote code is downloaded and executed. This bypasses code review and could introduce malicious code.Fix: Avoid downloading and executing remote scripts. Use trusted packages from PyPI/npm. If remote fetch is required, verify checksums and use HTTPS.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00056 | $0.04388 |
| Opus 5.5 | $0.00022 | $0.01755 |
| Sonnet 5.5 | $0.00011 | $0.00878 |
| Haiku 4.5 | $0.00006 | $0.00439 |
Grade F, and why
create-task scanned grade F with 4 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Reaches for credential fileshighPrivilege escalation
SSH keys, cloud credentials, git-credentials, .npmrc, /etc/shadow: reading these is how a config file becomes a credential leak.
Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`. Downloads and executes remote codehighSupply chain
curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.
apt-get update && apt-get install -y curl curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh Recursive force deletehighDestructive command
rm -rf with a variable or a broad path is one typo away from removing the wrong tree.
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/* Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
apt-get update && apt-get install -y curl Copies of this mod
3 near-identical copies found in the catalogue:
- create-task — 100% identical, 24 lines differ
- create-task — 94% identical, 12 lines differ
- create-task — 88% identical, 124 lines differ
How it starts
The opening of the file, as written. The whole thing — 483 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Guide the user through creating a new Harbor task end-to-end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part).
Step 1: Scaffold the task
harbor task init "<org>/<task-name>"
Useful flags:
--description "..."--author "Jane Doe <[email protected]>"(repeat for multiple authors)--no-pytest— skip the pytest test template (use if planning Reward Kit or custom verifier)--no-solution— skip solution/ directory--metadata-template path.toml— pre-populate task.toml
Produces:
<task-name>/
├── instruction.md # Task prompt for the agent
├── task.toml # Config and metadata
├── environment/Dockerfile # Container definition
├── solution/solve.sh # Reference solution (optional)
└── tests/test.sh # Verifier script
If the user wants a multi-step task (ordered steps with per-step
instructions, tests, and early stopping against a shared container), scaffold
the single-step layout first, then convert to the steps/ layout described in
the Multi-step tasks section below.
Step 2: Write instruction.md
This is the prompt the agent receives. Help the user write it clearly:
- State the goal concretely — what file to create, what behavior to produce
- Specify expected outputs — paths, formats, content
- Include constraints — language, tools, approach
- Don't leak the tests — describe what "done" looks like, not how you'll check it
Example (from the ssh-key-pair tutorial):
# SSH Key Pair Generation
Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.
Don't make them password protected.
Step 3: Build the environment
Edit environment/Dockerfile to install dependencies the task needs. The agent works
inside this container.
FROM ubuntu:24.04
WORKDIR /app
# Install what the task requires — NOT the solution
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago Changed a9ef48a93b06
- 13d ago Changed b3bd4e492d9a
- 24d ago Changed c9ef935a2dc1
- 1mo ago Changed fc8f76a0915b
- 1mo ago Changed · +7 lines ccd77ab4b4a0
- 1mo ago First seen · 476 lines · 56 tokens per session scan F 3619e134c568
create-task is a skill published in the GitHub repository harbor-framework/harbor (6,185 stars, last pushed today), licensed Apache-2.0. It adds 56 tokens to every session and 4,388 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it F with 4 findings (reaches for credential files, downloads and executes remote code, recursive force delete). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
agent-eval
Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent trajectories (tool correctness, completion), or picking an eval framework. NOT building the agent loop, tools or RAG…
evaluate-skill
Run, read, and diagnose a skill's Caliper eval — its success rate over k attempts, whether it fires, and whether it beats the agent without it. Use when the user wants to run, validate, interpret, or compare a skill's eval, or write an .eval.yaml spec whose tasks they have already decided.
writing-skills
Use when creating a new skill, editing an existing skill, or verifying skills work before deployment.
evals-clarify
Use when draft eval criteria need refining, clustering, and acceptance into the published goldset with an isolated holdout split (goldset.md + goldset.json).
evals-implement
Use when a published goldset is ready to become executable graders and configs — Python graders and metrics with auto-run unit tests verifying grader correctness.
evals-init
Use when standing up evals/{system}/ for the first time — scaffolds the EDD directory structure, picks PromptFoo or DeepEval by tech stack, and generates a security baseline.