lora

Using LoRA for Fine-tuning

LoRA (Low-Rank Adaptation) enables efficient fine-tuning by freezing pretrained weights and injecting small trainable matrices into transformer layers. This reduces trainable parameters to ~0.1% of the original model while maintaining performance.

Table of Contents

• Core Concepts
• Basic Setup
• Configuration Parameters
• QLoRA (Quantized LoRA)
• Training Patterns
• Saving and Loading
• Merging Adapters
• Best Practices
• References

Core Concepts

How LoRA Works

Instead of updating all weights during fine-tuning, LoRA decomposes weight updates into low-rank matrices:

W' = W + BA

Where:

• W is the frozen pretrained weight matrix (d × k)
• B is a trainable matrix (d × r)
• A is a trainable matrix (r × k)
• r is the rank, much smaller than d and k

The key insight: weight updates during fine-tuning have low intrinsic rank, so we can represent them efficiently with smaller matrices.

Why Use LoRA

Aspect	Full Fine-tuning	LoRA
Trainable params	100%	~0.1-1%
Memory usage	High	Low
Adapter size	Full model	~3-100 MB
Training speed	Slower	Faster
Multiple tasks	Separate models	Swap adapters

Basic Setup

Installation

pip install peft transformers accelerate

Minimal Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch

# Load base model
model_name = "meta-llama/Llama-3.2-1B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Configure LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 1,238,300,672 || trainable%: 0.28%