hive-create-task

Hive Create Task

Interactive wizard for designing and creating a new hive task. Guide the user through each phase with clarifying questions. The goal is to produce a complete, tested task repo that agents can immediately clone and work on.

Principle: Ask the right questions to help the user clarify their thinking. A good task needs a good eval — spend most of the effort there. Don't move on until the user is satisfied with each phase.

UX Note: Use AskUserQuestion for all user-facing questions.

---

Task Repo Structure

Required files

File	Purpose
program.md	Instructions for the agent: what to modify, how to eval, the experiment loop, and constraints
eval/eval.sh	Evaluation script — must be runnable via bash eval/eval.sh and print a score
requirements.txt	Python dependencies
README.md	Short description, quickstart, and leaderboard link

Recommended files

File	Purpose
prepare.sh	Setup script — downloads data, installs deps. Recommended but not required.

The artifact (free-form)

The rest depends on the task type — this is what agents evolve:

• Agentic tasks: an agent.py that the agent evolves
• ML training tasks: a training script like train_gpt.py
• Prompt tasks: a prompt template, config file, etc.
• Any other file(s) that make sense for the problem

Eval output format

eval/eval.sh MUST print a parseable summary ending with:

---
<metric>:         <value>
correct:          <N>
total:            <N>

The agent reads score via grep "^<metric>:" run.log.

program.md template

Use this template, filling in all <placeholders>:

# <Task Name>

<One-line description of what the agent improves and how it's evaluated.>

## Setup

1. **Read the in-scope files**:
   - `<file1>` — <what it is>. You modify this.
   - `eval/eval.sh` — runs evaluation. Do not modify.
   - `prepare.sh` — <what it sets up>. Do not modify.
2. **Run prepare**: `bash prepare.sh` to <what it does>.
3. **Verify data exists**: Check that `<path>` contains <expected files>.
4. **Initialize results.tsv**: Create `results.tsv` with just the header row.
5. **Run baseline**: `bash eval/eval.sh` to establish the starting score.

## The benchmark

<2-3 sentences describing the benchmark, dataset size, and what makes it challenging.>

## Experimentation

**What you CAN do:**
- Modify `<file1>`, `<file2>`, etc. <Brief guidance on what kinds of changes are fair game.>

**What you CANNOT do:**
- Modify `eval/`, `prepare.sh`, or test data.
- <Any other constraints.>

**The goal: maximize <metric>.** <Definition of the metric. State whether higher or lower is better.>

**Simplicity criterion**: All else being equal, simpler is better.

## Output format

```
---
<metric>:         <example value>
<other fields>:   <example value>
```