skill-benchmark-gate

Skill benchmark gate

Contract

Field	Bound contract
Trigger	Before shipping any skill change; diagnosing weak skill activation, regressions, or context cost; setting release gates for skill packs
Authority	Read and evaluate the skill; write only gate logs and issue artifacts; do not ship the change
Side effect	Runs evaluation loops; writes run-log entries; opens follow-up issues on GitHub or equivalent tracker
Done	A reproducible PASS or FAIL record with case results, deltas, context cost, and failure issues

Inputs

All inputs are required and must be explicit. No defaults are inferred from the procedure.

1. Changed skill and comparison revision: the skill slug, the diff or branch under test, and the baseline revision to compare against.
2. Explicit universal activation case set: the complete list of cases that must all show activation greater than 0% with the skill enabled. Every case must be named; no case is implied.
3. Explicit critical regression case set and baseline: the list of critical scenarios and their recorded baseline results (pass/fail and context token count per scenario) from the comparison revision. Every scenario must be named with its baseline; no scenario is implied.
4. Explicit context-budget ceiling: the maximum total context tokens the skill may load, stated as a number.
5. Explicit iteration budget: the maximum number of evaluation iterations the gate may run, stated as a number.

Procedure

1. Validate all gate inputs fail-closed. Confirm every required input is present and non-empty. If any input is missing or incomplete, fail immediately with missing gate input and name the missing input. Do not proceed to evaluation. Done when: all five required inputs are confirmed present and explicit.
2. Run activation cases. Execute each case in the universal activation set with the skill enabled. Record whether the skill fired (activated) or missed for each case. Done when: every case in the universal activation set is executed and its activation result is recorded.
3. Run baseline and changed regression cases. Execute each critical regression scenario against the comparison revision (baseline) and against the changed skill. Record pass/fail and context token count per scenario for both. Done when: every critical regression scenario is executed against both baseline and changed skill, and results are recorded.
4. Measure context cost. Sum the total context tokens loaded by the skill across all executed cases. Compare against the explicit context-budget ceiling. Done when: the total context cost is measured and compared against the ceiling.
5. Emit PASS or FAIL and file one issue per failure. Apply the gate criteria:
   • Universal activation: every case in the universal activation set must show activation greater than 0%. Any 0% activation is a gate failure.
   • Regression deltas: compare each critical scenario's changed result against its recorded baseline. Any negative delta (baseline pass to changed fail, or increased token count) is a gate failure.
   • Context budget: total context tokens must not exceed the explicit ceiling. An overrun is a gate failure.
   • Iteration budget: if the iteration budget is exhausted before all cases are executed, the gate fails with iteration budget exhausted. For each gate failure, open a follow-up issue with the failure class, scenario or case ID, observed value, and expected threshold. Write the run-log entry: timestamp, skill slug, change reference, case results, deltas, context cost, gate verdict, and filed issue references. Done when: the verdict is emitted, one issue is filed per failure (or none if the gate passed), and the run-log entry is written.