build-task-benchmarks

Building task benchmarks

A per-command smoke test proves a command reaches its backend. It cannot tell you whether a user can get a job done, because a job is many commands in sequence, and everything that makes sequences hard — navigation, handles passed between turns, the context a turn lands in — is exactly what a per-command test removes.

Build the benchmark in two tiers. Never skip the first one.

Tier	Shape	Answers
Conversations	3-15 turns, one errand, usually one workspace	Does each capability work in its natural setting?
Tasks	20-100 steps, several conversations chained, many contexts	Does a real job complete?

The tiers are not two views of the same thing. Tier 1 is your unit-test layer and the block library tier 2 is assembled from. That is what makes a tier-2 failure attributable: every block is independently green, so a step that only fails at length is a length problem, not a command problem. If you write tier 2 first you will spend your time debugging whether the command or the chain is broken.

Tier 1 — conversations as unit tests

Structure

A conversation is a name, a purpose, and an ordered list of turns. A turn is what the user says, the command that should handle it, its parameters, and optionally what to capture from the response and which context it should land in. Full dataclass and runner skeleton: reference.md.

Four rules make the difference between a real harness and a loop over command calls:

1. Navigation is real. The context a turn lands in is the context the next turn runs in. Do not reset between turns. A conversation that only works because each turn was handed a fresh context is not testing anything a user will do.
2. Handles come out of the rendered response. Parse the uid a later turn needs from the text an earlier turn produced, not from internal state. The rendered response is all an agent ever sees, so if a listing does not expose its handles legibly, that is a defect this suite should catch.
3. Run the validation hook. Signature.validate_extracted_parameters is part of the runtime path, and normalising rules write back onto the parameter object the command body then reads. Skipping it tests a call the runtime never makes. Return a validation failure as a soft outcome, not an exception — the runtime hands the message back for correction, so a bad call is still a conversation.
4. Writes are opt-in and paired. Every write turn gets a restore turn, so a default run is net-neutral. Anything genuinely destructive is validated (its input object is constructed) and never sent.