os-lab-scorer

Lab Scorer (os-lab-scorer)

One locked check for one change. It never loops: the loop belongs to os-lab-loop, which calls this skill once per change. Two modes share the same gate: check (score a change someone else proposed) and single pass (RED-GREEN-REFACTOR on one skill).

Contents

• Critical constraints
• Quick start
• Workflow
• Verification
• References

Critical constraints

1. Locked evaluator: never modify evaluate.py, eval_runner.py or evals/evals.json to make a score pass. Only the target SKILL.md changes.
2. Objective verification: never simulate routing or judge by impression. Every verdict comes from running evaluate.py.
3. One variable per change: reject bulk rewrites; the gate cannot attribute a score change to several edits.
4. Holdout gate: if base score rises but holdout score falls, force DISCARD (overfitting). A run with no holdout set is incomplete.
5. F1 guard: never disable the f1 >= baseline_f1 check; it is the only defense against keyword stuffing.

Quick start

Baseline a target once, then score each change:

python scripts/evaluate.py --skill <path/to/target-skill> --baseline --desc "initial baseline"
python scripts/evaluate.py --skill <path/to/target-skill> --desc "what changed"
echo $?   # 0 = KEEP, 1 = DISCARD (reverted), 2 = path error, 3 = tampered environment

Pass the skill folder, not a file. State lives in the target: evals/evals.json, evals/results.tsv and its optimization program file.

Workflow

Mode 1: Check one change

1. Confirm the proposal changes one variable and write it to the target SKILL.md.
2. Run evaluate.py; read the exit code and the new row in evals/results.tsv.
3. Apply the holdout gate (see overfitting-gate.md).
4. DISCARD: confirm SKILL.md was restored and report score, delta and F1. KEEP: commit the single change (keep: score=<s> f1=<f1> <desc>).
5. Return the verdict to the caller. Exit codes 2 and 3 are not verdicts; see troubleshooting.md.