accelerate

Accelerate: Distributed Training Made Easy

Overview

Hugging Face Accelerate provides a simple API for scaling PyTorch training across multiple GPUs, TPUs, or CPUs with minimal code changes. Apply this skill for distributed training, mixed precision, gradient accumulation, and seamless hardware scaling.

When to Use This Skill

This skill should be used when:

• Training on multiple GPUs
• Using mixed precision (FP16/BF16) for faster training
• Running on TPUs
• Implementing gradient accumulation for large batches
• Scaling from laptop to cloud seamlessly
• Integrating with DeepSpeed
• Converting standard training loops to distributed
• Handling device placement automatically

Quick Start

Basic Setup

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from accelerate import Accelerator

# Initialize accelerator
accelerator = Accelerator()

# Auto-handles device placement, mixed precision, and distributed training
model = nn.Linear(10, 10)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
train_loader = DataLoader(...)  # Your data

# Prepare everything for accelerator
model, optimizer, train_loader = accelerator.prepare(
    model, optimizer, train_loader
)

# Training loop - just add accelerator.backward()
for batch in train_loader:
    optimizer.zero_grad()
    outputs = model(batch)
    loss = outputs.sum()
    accelerator.backward(loss)
    optimizer.step()

Using from Scratch

# Convert existing training script with minimal changes
from accelerate import Accelerator

accelerator = Accelerator(
    mixed_precision="fp16",  # or "bf16"
    gradient_accumulation_steps=2,
    log_with="tensorboard",
    project_dir="./logs"
)

# Wrap model, optimizer, dataloader
model, optimizer, dataloader = accelerator.prepare(
    model, optimizer, dataloader
)

# Training loop
for batch in dataloader:
    with accelerator.accumulate(model):
        outputs = model(batch)
        loss = loss_fct(outputs, targets)
        accelerator.backward(loss)
        optimizer.step()
        optimizer.zero_grad()