trapstreet-task-scaffold

trapstreet-task-scaffold

Scaffolds a new task directory in trapstreet-tasks and guides the design decisions that make a task actually good -- discriminating, hard to game, legally sound, and consistent with a real ground-truth pipeline. Sister skill to trapstreet-solution-scaffold, which does the reverse (build a solution against an existing task).

Read this first, honestly: unlike solution scaffolding, task design is not fully mechanizable. The file layout, manifest contracts, and aggregation logic are the same every time and the scaffold script writes them for you. Whether the task is actually good -- whether it measures something real, whether it resists gaming, whether the ground truth is sound -- depends on understanding the specific agent/skill/domain being tested, and that part is an interview + judgment call, not a template fill.

And the second thing to know: intuitions about what makes a task hard are unreliable, so the workflow below is built to find that out early and cheaply -- probe one question before authoring a set, and never conclude from a single run. references/difficulty-design.md and references/calibration.md are the two files that decide whether the finished task discriminates; the rest is craft around them.

Ground rules

• Never push to the shared task repo, and never register/publish a task on trapstreet.run, without the user's explicit go-ahead on that specific push/publish -- same weight as trapstreet-solution-scaffold's submit rule. Agreeing to earlier steps (case design, scoring logic) is not consent to publish; ask again at that specific moment.
• Default to local-only whenever the legal/IP question (interview step 5) is unresolved. Build and test the task fully -- nothing about that requires a public remote -- but don't git push until the question is actually answered (see references/legal-ip-checklist.md).

Before writing anything: interview

1. What does the agent/skill actually do, concretely? Not "a code review skill" but "given a diff, flags likely bugs with a file/line/ description." The task's I/O contract should mirror the real thing this tool is used for -- don't design a task that only tests a narrow slice of what the tool claims to do, or one so different from its real usage that good performance here doesn't predict good performance there.
2. What does "correct" mean, concretely, and who would disagree? If two competent humans could reasonably disagree on whether an answer is right, that's a sign the scoring needs either a very carefully curated rubric or a different, more objective framing of the task.
3. What is supposed to make this hard, and is that thing real? Answer in the two quantities that predict the score: H*, the minimum number of effective actions the task requires, and s, the layers of nested sub-goals and conditional branches. Performance falls off non-linearly in s with a sharp knee; the intuitive answers (harder arithmetic, defects a human would be slow to spot, capability gates a shell can synthesise) sit on the flat part and moved a bare harness not at all. Read references/difficulty-design.md before answering -- it is the difference between a task that discriminates and one everyone passes. Then answer a third question it raises: what is in the material? H* and s describe the procedure; they say nothing about whether the answers are sitting in the document as plain text. Three probe rounds on one task raised depth and horizon and moved a 20/20 ceiling not at all; changing what the document contains broke it on the first attempt. And if the task puts a set of options in front of the solver -- a tool menu, a skill catalog, retrieval candidates -- read "When the task varies a candidate set" in the same file first. Accuracy at N=8 and N=26 are not comparable without a chance correction or a size-matched control; distractors picked by hand make confusability a claim about the author rather than a property of the task; and a control arm matched on the countable thing can be unmatched on the thing that actually fires. All three shipped in one task before being caught, and the third was about three quarters of its headline number. Then ask the mirror question -- what will make a bad solution score badly? -- and read references/making-a-task-discriminate.md, which is where four case sets that separated nothing are written up. Its rule is that disorder is recoverable and absence is not: a capable model repairs a garbled input, so grading how well something survived grades the repair. And verify the failure is actually present before authoring a single case -- one task shipped 27 cases and returned 270 scores of 1.0.
4. Where does ground truth come from? Computed from a seed (no answer for anyone to get wrong, and leakage is impossible by construction), real historical data (leakage risk, but credible), or hand-authored (no leakage risk, but needs real effort to feel authentic)? Read references/ground-truth-sourcing.md before deciding -- this is one of the highest-leverage decisions in the whole task, and the computed option is under-used.
5. Does any candidate source material raise a legal/IP/liability question? Read references/legal-ip-checklist.md and answer its questions explicitly before writing a single case into gold.cases.json. If the answer is unclear, default to building the task locally (gitignored) and resolve the question before ever pushing it to a public remote -- not after.
6. How many cases, and how are they organized into categories/tags? Enough to give real signal (a handful of cases barely discriminates anything), but every case should be worth its inclusion -- don't pad the count with near-duplicates of an already-covered pattern. "Near- duplicate" is measurable rather than a matter of taste, so write the per-capability budget down here and check it against the runs later -- "Budget cases by capability" in references/difficulty-design.md has the allocation rule and the two checks, one of which passes on a set whose 22 items span four independent directions. Plan on sourcing roughly three times what you intend to ship: SWE-bench Verified discarded 68.3% of naturally sourced candidates under review, and the probe below will discard some of yours.