model-fine-tuning

Model Fine-Tuning

Purpose: Adapt pre-trained foundation models to domain-specific tasks through parameter-efficient and full fine-tuning techniques.

---

When to Use This Skill

• Adapting an LLM to follow domain-specific instructions or style
• Improving model accuracy on specialized tasks (classification, extraction, generation)
• Reducing inference cost by distilling a large model into a smaller one
• Building training data pipelines for fine-tuning datasets
• Choosing between fine-tuning approaches (LoRA, QLoRA, full, distillation)

Prerequisites

• Base model selection (open-source or API-based)
• Training dataset (instruction pairs, labeled examples, or preference data)
• GPU compute (local or cloud)

Decision Tree

Need model customization?
+- Task-specific behavior (format, style, domain)?
|  +- Small dataset (<1K examples)?
|     +- Use few-shot prompting first -> Good enough? -> Done
|     +- Not enough? -> LoRA fine-tuning (efficient, low data)
|  +- Medium dataset (1K-50K examples)?
|     +- Use LoRA or QLoRA -> Most cost-effective
|  +- Large dataset (50K+ examples)?
|     +- Full fine-tuning (if budget allows)
|     +- Or LoRA with longer training
+- Need smaller/faster model?
|  +- Knowledge distillation from large to small model
|  +- Prune and quantize after fine-tuning
+- Need alignment (safety, helpfulness)?
|  +- DPO (Direct Preference Optimization) on preference pairs
|  +- RLHF if reward model available
+- Using API-based model (OpenAI, Azure)?
   +- Use provider fine-tuning API
   +- Prepare JSONL format per provider spec

---

Fine-Tuning Approaches

Approach	Parameters Trained	VRAM Required	Data Needed	Best For
Full Fine-Tuning	All	Very High (80GB+)	10K-1M+	Maximum performance, if budget allows
LoRA	Low-rank adapters (0.1-1%)	Medium (16-24GB)	1K-50K	Most common; great cost/quality tradeoff
QLoRA	Quantized base + LoRA	Low (8-16GB)	1K-50K	Constrained GPU budget
Prefix Tuning	Soft prefix tokens	Low	500-10K	Simple task adaptation
Adapter Tuning	Small adapter layers	Low-Medium	1K-50K	Multi-task with shared base
Knowledge Distillation	Student model (all)	Medium	Large unlabeled + teacher	Deploying smaller models
DPO	Full or LoRA	Medium-High	5K-50K preference pairs	Alignment, safety, style