fine-tune-readiness

Fine-Tune Readiness

1 · Purpose

Fine-tuning is the last resort in a ladder of cheaper levers, this skill makes the team climb the ladder in order, and gates the dataset before a single training run.

2 · When to use

• Whenever "we should fine-tune" is proposed.
• Before any SFT/LoRA/DPO run, to gate the dataset.
• When a fine-tuned model underperforms and nobody knows why (usually: the data).

3 · Inputs

• The task spec with metric and gate (spec-first-ai-feature).
• An eval that can score the behavior change (eval-first-development).
• The failure log from the current prompt/RAG system, clustered (error-analysis-50).

4 · Procedure

1. Climb the ladder in order, recording the score each rung achieves: (a) better prompting → (b) few-shot examples → (c) RAG with the right documents → (d) fine-tuning. Quote the score at each rung; do not skip rungs.
2. Name the failure class fine-tuning is meant to fix. Fine-tuning fixes behavioral classes, format, tone, style, domain vocabulary, consistent refusal/compliance patterns. It does not fix missing facts (that is RAG) or weak reasoning (that is a bigger model or decomposition).
3. If the failure class is knowledge or freshness, STOP: use RAG. If it is format or tone and rungs a-c are exhausted, proceed.
4. Gate the dataset before training:
   • ≥ 200 examples for SFT format/behavior shifts (1,000+ for real moves); preference pairs for DPO.
   • Every example reviewed or generated against a written standard.
   • No PII unless the compliance sign-off exists in writing.
   • A held-out slice (≥ 10%) that training never sees.
5. Choose the method: LoRA/QLoRA first (small adapter, reversible, cheap); full fine-tune only when adapters demonstrably cannot move the metric. DPO only after SFT, when you have preference pairs.
6. Compute the hardware budget (local-model-fit) or the cloud training cost before launching. Write the number down.
7. Train, then score on the held-out slice and the golden set. Compare against the best prompt/RAG rung, not against the untrained base model alone.
8. Ship only if the fine-tune beats the best non-trained rung by a margin that justifies the serving and maintenance cost. Record the decision and the numbers.