distill-classifier

Distill Classifier

Most production LLM calls are not agents — they are classifiers: tag, route, score, extract. This worker replaces a frontier classifier with a fine-tuned open-weight student using teacher-as-labeler hard-label distillation: an ensemble of teachers votes labels onto unlabeled rows, the student trains on the consensus, and a gated verdict decides whether it ships. It does not cover reasoning or tool-calling workloads — for those use ../compare-trajectories/SKILL.md and the training rungs it feeds.

Decision Gate

Right move when the task is knowledge-bound classification (domain label boundaries, not instruction-following), call volume justifies a ~$5–15 experiment, and either the teacher's logits are closed (hard labels are all you can get) or several teachers are available. Wrong move when prompt optimization hasn't been tried — run ../optimize-workload/SKILL.md (GEPA) first; it has moved classifiers +8pp accuracy with no weight update, and SFT is only warranted after it plateaus against a measured baseline.

Safety Gates

• Do not upload source files, prompts, traces, labels, or datasets unless the developer explicitly approves that exact action in the current thread. Teacher-labeling calls send rows to providers — name the row count and get approval before the sweep.
• Frozen splits before any labeling: train/dev/holdout from ../capture-evidence/SKILL.md, with a leakage check across splits. Teachers label train only; dev is for checkpoints; holdout is scored once, at the end.
• Never report accuracy on imbalanced classes — macro-F1 and per-class recall are the promotion metrics; accuracy hides class collapse.
• Run the confound ablations (see reference.md) before attributing any lift to training.

Flow

1. Baseline the teachers. Score ≥2 (prefer 3) frontier/strong models on a balanced labeled sample (≥200 rows/class). Record per-model macro-F1 and per-class recall. Pick the N best teachers — mean teacher quality predicts student quality; teacher diversity does not (confirmed null, see reference.md).
2. Try the no-weight rung. GEPA on train rows only. If macro-F1 lands within ~3pp of the best-teacher ceiling, stop — ship the prompt.
3. Consensus-label train. Each teacher labels every train row; majority vote per row (per label for multi-label). Keep ≥80% of rows: volume beats purity — strict confidence filtering measurably hurts (−8pp macro-F1 in the controlled comparison). Set the split-vote rows aside as the disagreement set.
4. Build a failure-directed corpus. Run the student zero-shot on train; build the SFT set as roughly 60–70% student-miss rows (consensus label the student got wrong) plus 30–40% unanimous correct rows balancing the minority class to ~50/50. Targeting residual failures beats a larger clean-unanimous corpus.
5. LoRA SFT. One epoch, conservative LR, and LoRA rank ≥64 — r32 and below cannot override the base model's class prior and collapses minority recall (the single most load-bearing hyperparameter measured; details in reference.md). Train locally via ../local-distillation-lab/SKILL.md or export the JSONL for a hosted SFT job (approval-gated; size it with ../plan-hosted-run/SKILL.md).
6. Validate, then verdict. On dev: macro-F1 vs best teacher, per-class recall ≥50% everywhere, schema-validity 100% for structured output. Then one holdout pass and a four-way verdict:
   • PROMOTE — beats the bar at ≤ the cost target → route it (../use-understudy-gateway/SKILL.md).
   • SHADOW-TEST — passes overall but a critical class is marginal.
   • COLLECT-MORE-DATA — holdout too thin (<~120 rows) to decide.
   • STOP — far below bar after GEPA and SFT; do not iterate hyperparameters more than twice — the ceiling is the data or the task.
7. Escalation (optional). If the student plateaus exactly on the disagreement set and an open teacher with logits is available, soft-target distillation on that boundary slice is the next rung (measured +7.9pp on ambiguous classes; see reference.md).