experiment

Experiment

Use this skill for the main evidence-producing runs of the quest. The goal is to turn one selected route into one trustworthy measured result with the smallest valid amount of execution.

Match signals

Use experiment when:

• a baseline is accepted
• an idea has been selected
• the evaluation contract is explicit
• the quest is ready for implementation and measurement rather than framing, route selection, or writing

Do not use experiment when:

• the baseline gate is unresolved
• the idea stage still has unresolved tradeoffs
• the main need is writing or follow-up analysis rather than a main run
• the real problem is still route choice, baseline recovery, or open-ended optimization rather than one bounded measured run

One-sentence summary

Turn one selected route into one trustworthy measured result with the smallest valid amount of execution, then record and route from the evidence.

Quick workflow

• Recover the selected idea, accepted baseline, metric contract, and current workspace before implementation.
• Keep the selected idea summarized in 1-2 sentences, then write a minimal code-change map before touching broad code.
• Define the null hypothesis, alternative hypothesis, research question, research type, research objective, experimental setup, experimental results, experimental analysis, and experimental conclusions as the run matures.
• Run only the checks needed to maximize valid evidence per unit time and compute.
• Use equivalence-preserving efficiency upgrades when they preserve baseline comparability; For comparison_ready, verify-local-existing, attach, or import should usually beat full reproduction.
• If an efficiency change affects baseline comparability, treat it as a real experiment change.
• Prefer one clean implementation pass and one real run over repeated half-runs when the route is already concrete.
• Implement according to the current PLAN.md; revise the plan before changing the route.
• implement according to the current PLAN.md
• Extra metrics are allowed, but missing required metrics are not.
• extra metrics are allowed, but missing required metrics are not
• If a useful non-canonical metric appears, record it as supplementary output rather than replacing the canonical comparator.
• In algorithm-first work, experiment is the execution surface of optimize, then results return to optimize or decision for frontier review.
• End with a concise 1-2 sentence outcome summary, evaluation_summary, claim_update, baseline_relation, failure_mode, and next_action.