fine-tuning

Fine-Tuning

Production authority on adapting large language models: parameter-efficient fine-tuning with LoRA/QLoRA, full supervised fine-tuning, preference alignment via DPO and RLHF, quantization for deployment, and systematic evaluation. Use this skill when training or adapting any LLM beyond prompting, including domain adaptation, instruction following, and RLHF pipelines.

Core Concepts

LoRA vs QLoRA vs Full Fine-Tune

LoRA (Low-Rank Adaptation) injects trainable rank-decomposition matrices into attention layers, leaving base weights frozen. A rank-16 LoRA on a 7B model trains ~8M parameters instead of 7B — fits in ~24 GB VRAM. QLoRA adds 4-bit NF4 quantization of the frozen base weights (via bitsandbytes), enabling 7B fine-tuning on a single 16 GB GPU. Full fine-tuning is reserved for fundamental domain shifts where LoRA rank capacity is genuinely insufficient, or when you have 8+ A100s.

PEFT + Hugging Face Trainer (LoRA SFT)

from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset
import torch

MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"  # critical for causal LM training

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,   # bf16 > fp16 for training stability
    device_map="auto",
    attn_implementation="flash_attention_2",  # 3-5x memory savings
)

lora_config = LoraConfig(
    r=16,                    # rank — higher = more capacity, more VRAM
    lora_alpha=32,           # scaling factor; effective lr scales as alpha/r
    target_modules=[         # target all attention projections
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",  # include MLP for better perf
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.52

dataset = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

args = TrainingArguments(
    output_dir="./llama3-8b-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    eval_strategy="epoch",
    load_best_model_at_end=True,
    report_to="wandb",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    dataset_text_field="text",      # or use formatting_func for chat templates
    max_seq_length=4096,
    packing=True,                   # pack short sequences to fill context window
)
trainer.train()
model.save_pretrained("./final-lora-adapter")