model-retuning

Model retuning

Contract

Inputs

• Target model or symptom: required. Names what the corpus degrades on.
• Corpus path: optional, defaults to ./skills. The skill directories to retune.
• bar:<n>: required. The number of consecutive clean runs the corpus must clear, registered in writing before any change exists.
• A benchmark harness that can A/B two builds of the corpus: required. Refuse without it.

Procedure

1. Measurement gate. Confirm all three before any work: a run archive (or harness that produces one) carrying per-run tool-call traces, terminal markers, token counts, and final messages; a build selector that points a run at a specific corpus checkout so two builds are comparable under one runner; and a repeatable task the corpus executes end to end. Name whichever is missing and stop. Do not fall back to a static audit and present it as retuning. Done when: all three substrate requirements are confirmed or the missing one is named and the run stops.
2. Mine the archive before spending a run. Zero model cost: read files already on disk. Derive a phase-marker map from the corpus's own spine — one observable marker per phase (a tool call or a file that exists, never a phrase in prose) — before writing an extractor. Extract one row per run by script, not by hand, with at least: run id, model, settings, session and child ids, ordered phase trace, terminal marker present, output tokens, wall clock, helper dispatch count, max parallel dispatch, and the verbatim final message. Derive task_done (deliverable exists) and process_followed (every required phase appears in spine order) independently; never let one imply the other or let the marker stand in for either. Score outcomes first-match-wins with broken > halt > wrong-result > task-done-no-process > success. Done when: every archived run has an extracted row with both metrics scored independently.
3. Establish the noise floor. Run the harness against two identical copies of the corpus, same commit on both sides; whatever difference appears is the floor every later claim must clear. Register the bar now, in writing, before any change exists — a bar chosen after seeing results is not a bar. Run arms serial within an arm and interleaved across arms with nothing else running; contention destroys wall-clock-derived metrics and can manufacture a timeout that reads as a halt, so a contended run cannot support a latency claim and cannot be counted as a halt. Done when: the noise floor is measured, the bar is registered in writing, and no corpus change exists yet.
4. Audit the corpus adversarially. One agent per skill directory proposes cuts; a second per skill defends the existing prose. The two passes require independent contexts. If the host exposes no way to run them as separate agents, report that as a blocker and stop — do not argue both sides in one context and present the result as an audit. Budget the defense as first-class work: the removals are the phase's product as much as the proposals. Done when: both passes complete in independent contexts or the host limitation is reported as a blocker.
5. Cut in surgical passes. One problem class per pass, no bundling. Fan out by disjoint file ownership, never by item: one agent owns one skill directory and applies the class everywhere inside it, so ownership is a checkable filesystem partition with every path in exactly one manifest row. Discover byte-identical duplicated assets before dispatch and assign every copy to exactly one owner who propagates the edit to all copies in the same pass; list them as forbidden for everyone else. When a rewrite's strings cross-reference each other, author the canonical mapping (old string to new string, exact) serially first, then fan out verbatim application; an uncovered occurrence is a contract gap to resolve serially, not a variant to improvise. Reconcile every cross-reference before the pass closes. Done when: one problem class is applied across all owned paths and every cross-reference is reconciled.
6. Measure, then let the failure choose the next fix. A failure that moves to a later phase is progress and names the next target; a failure at the same site means the fix missed; a run that completes the task while skipping the workflow is a different defect than a halt and only shows up if step 2's two metrics stayed separate. Audit the phases the instrument cannot reach: a probe that skips a phase can never fail in it, so a green streak certifies only what it exercised. Loop steps 5 and 6 until the registered bar clears, then stop. Report the paths that remain unmeasured and what would be needed to measure them; one clean run proves nothing. Done when: the registered bar clears or the run reports the specific claim it could not support and the unmeasured paths.
7. Ship. Commit each pass separately with its own message so history says which change was made and why. Keep the measurement artifacts. Write the finding where the next person will hit it: the mechanism, before and after, the measured numbers, and the hypotheses that died — the dead ends are what stops the next attempt re-running them. Done when: every pass is committed separately and the finding is written at the point of next impact.