rewardkit

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards.

Setup in a Harbor task

Put criteria alongside test.sh in the task's tests/ directory:

tests/
├── test.sh
├── checks.py         # programmatic criteria
└── judge.toml        # optional LLM/agent judge

tests/test.sh:

#!/bin/bash
uvx --from 'harbor-rewardkit==0.1.*' rewardkit /tests

This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json. Defaults match Harbor's conventions — no extra config needed.

If judge criteria need API keys, pass them through task.toml:

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean-room checks should not be available to the agent:

[environment]
network_mode = "no-network"   # Agent env baseline — offline during agent.run()

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"     # Verifier env baseline — LLM judge API calls
docker_image = "python:3.12-slim"

In shared mode, the verifier runs in the agent container and inherits [environment].network_mode. Put [verifier].network_mode only when verify() needs different network access than the agent phase (a phase override, not a baseline). If agent and verifier need different baselines without runtime switching, use environment_mode = "separate" and set [verifier.environment].network_mode.

Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no-network.

In separate mode, tests/ is the verifier image build context and must provide /tests/test.sh at runtime; Harbor does not upload tests/ into the running verifier container.