pytorch

PyTorch

PyTorch is the foundational deep learning framework for research and production, featuring dynamic computational graphs, torch.compile graph optimization, and distributed GPU training.

When to Use

• State-of-the-Art Deep Learning Research & Production: Training vision, NLP, audio, and generative AI models.
• Model Compilation with torch.compile(): Accelerating model inference and training via TorchDynamo and Triton kernels.
• Distributed Training (DDP / FSDP): Scaling models across multi-GPU and multi-node clusters.
• Hardware Acceleration: Training and executing on NVIDIA CUDA, Apple Silicon MPS, and AMD ROCm.

Quick Start

import torch
import torch.nn as nn

# Define simple neural network
class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(20, 64),
            nn.ReLU(),
            nn.Linear(64, 2)
        )

    def forward(self, x):
        return self.net(x)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = MLP().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

Core Concepts

Modern Neural Network Module with torch.compile

Defining architectures and compiling with TorchDynamo:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerClassifier(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, num_classes: int):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=4, dim_feedforward=256, batch_first=True
        )
        self.transformer = nn.TransformerEncoder(self.encoder_layer, num_layers=2)
        self.fc = nn.Linear(embed_dim, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        embedded = self.embedding(x)
        encoded = self.transformer(embedded)
        pooled = encoded.mean(dim=1) # Global average pooling
        return self.fc(pooled)

device = torch.device("cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu"))
model = TransformerClassifier(vocab_size=10000, embed_dim=128, num_classes=5).to(device)

# PyTorch 2.x JIT compilation for Triton / GPU optimization
if torch.cuda.is_available():
    compiled_model = torch.compile(model)
else:
    compiled_model = model