tune-loop

tune-loop — the self-improving system

The other four skills build one model. tune-loop closes the loop: serve → log → collect feedback → curate → experiment across architectures × methods → promote only what measurably beats the champion → repeat. The "AI system" stops being a model and becomes the loop; what sits inside (a single model, a cascade, an agent, a deterministic workflow) is just the current champion, which the loop is free to replace when evidence says so.

This is champion/challenger from classical MLOps, generalized to compound-AI architecture search, driven by a Monitor–Analyze–Plan–Execute (MAPE) control loop. It deliberately reverses tunelab's v1 "no autonomous hill-climbing" non-goal — and the disciplines below are what make the reversal teaching-grade rather than AutoML slop.

Step 0 — read the system state before anything

On invocation, read the project's system/ dir:

• descriptor.json — the current champion architecture (see schema below).
• predictions.jsonl — the append-only flywheel log (tune-data's flywheel.py schema).
• EXPERIMENT-LOG.md + rounds/*/ — prior rounds, promotions, and the eval slices each consumed. Never reuse a consumed slice.

The system descriptor (architectures as data)

A small versioned JSON makes architectures enumerable, comparable, reproducible:

{ "version": 3, "kind": "cascade",
  "components": [
    {"id": "t1", "model": "lr", "train": "embeddings+logreg", "calibrate": "isotonic"},
    {"id": "t2", "model": "qwen3-4b", "train": "qlora-sft", "conf": "token-margin"},
    {"id": "t3", "model": "frontier", "prompt": "knn-fewshot"}],
  "routing": "t1 ->[cal_conf<0.43] t2 ->[cal_conf<0.60] t3",
  "thresholds": {"t1": 0.43, "t2": 0.60} }

kind ∈ {single, cascade, workflow}. Examples: single: FT-SLM(qwen3-4b, sft) · workflow: injection-gate → router → {cheap, frontier}. The descriptor is the unit the loop mutates and the eval adjudicates.

The MAPE round (the crank)