ml-experiment-tracker

ML Experiment Tracker

A skill for planning, executing, and tracking machine learning experiments with full reproducibility. Covers experiment design, hyperparameter management, metric logging, model versioning, and comparison across runs to support rigorous ML research.

Overview

Machine learning research involves running dozens or hundreds of experiments with varying architectures, hyperparameters, data splits, and preprocessing pipelines. Without systematic tracking, it becomes impossible to reproduce results, compare configurations, or identify which changes actually improved performance. This skill provides a structured methodology for experiment management that aligns with academic standards for reproducible ML research.

The approach is framework-agnostic but demonstrates integration with MLflow, Weights & Biases, and plain file-based logging. It emphasizes the practices needed for publications: complete hyperparameter documentation, statistical significance testing across runs, and artifact management for model checkpoints and evaluation outputs.

Experiment Design Framework

Defining an Experiment Plan

Before writing any training code, document the experiment plan:

# experiment_plan.yaml
experiment:
  name: "transformer-sentiment-analysis-v3"
  hypothesis: "Adding relative positional encoding improves F1 on long reviews (>512 tokens)"
  dataset:
    name: "imdb-extended"
    version: "2025.1"
    splits: {train: 0.8, val: 0.1, test: 0.1}
    stratify_by: "label"
    random_seed: 42

  baselines:
    - name: "bert-base-uncased"
      checkpoint: "bert-base-uncased"
    - name: "roberta-base"
      checkpoint: "roberta-base"

  variables:
    independent:
      - positional_encoding: ["absolute", "relative", "rotary"]
    controlled:
      - learning_rate: 2e-5
      - batch_size: 32
      - max_epochs: 10
      - early_stopping_patience: 3
      - optimizer: "AdamW"
      - weight_decay: 0.01

  metrics:
    primary: "f1_macro"
    secondary: ["accuracy", "precision_macro", "recall_macro", "loss"]
    report_at: ["best_val", "final"]

  compute:
    gpus: 1
    estimated_time_per_run: "45min"
    total_runs: 9  # 3 encodings x 3 seeds

  seeds: [42, 123, 456]