deep-learning-pytorch

Deep Learning with PyTorch

Overview

Deep learning bugs rarely crash: a softmax before the loss, dropout left on during evaluation, normalisation statistics computed on the test set — the loss still goes down. Structure the code so each piece is a small function you can test on CPU in a second, and prove the wiring by overfitting one tiny batch.

Core principle: the eager CPU path is the tested path; GPUs, torch.compile and mixed precision are configuration on top of it, never a different code path.

1. Structure

models/net.py        nn.Module, no data loading, no device logic
data/dataset.py      Dataset returning tensors; normalisation stats passed in, fitted on train only
train/loop.py        train_one_epoch(), evaluate(), fit() driven by a config object
pipelines/train.py   entrypoint: config → seed → data → fit → checkpoint → log

def train_one_epoch(model, loader, optimizer, loss_fn, device) -> float:
    model.train()
    total = torch.zeros((), device=device)
    seen = 0
    for xb, yb in loader:
        xb, yb = xb.to(device, non_blocking=True), yb.to(device, non_blocking=True)
        optimizer.zero_grad(set_to_none=True)
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        total += loss.detach() * len(xb)
        seen += len(xb)
    return (total / seen).item()

@torch.inference_mode()
def evaluate(model, loader, device) -> tuple[torch.Tensor, torch.Tensor]:
    model.eval()
    logits, targets = [], []
    for xb, yb in loader:
        logits.append(model(xb.to(device)).cpu())
        targets.append(yb)
    return torch.cat(logits), torch.cat(targets)

Accumulating loss.detach() avoids a GPU sync per step that loss.item() forces.

2. Classic bugs