ai-ml-engineering

Overview

ML engineering failures are silent and delayed. A model that scores well on the benchmark can fail badly in production. This skill enforces the practices that catch these failures before they reach users: proper evaluation harnesses, data leakage detection, distribution shift monitoring, and safety checks.

When to Use

• Before training or fine-tuning a model
• Before deploying a model to production
• When integrating a third-party LLM API
• When evaluating model quality
• When debugging unexpected model behavior

Process

Step 1: Define the task and success metric precisely

Before any code: what is the exact prediction task? What metric proves the model is good enough? What metric proves it is safe enough? Document these as your evaluation contract.

Step 2: Establish the baseline

Compute a simple baseline (majority class, rule-based system, GPT-4 zero-shot). Your model must beat this baseline by a meaningful margin to justify the complexity.

Step 3: Audit the training data

• Check for data leakage (test set information in training set)
• Check for label quality (sample 100 examples manually)
• Check for demographic skew (does the dataset represent production distribution?)
• Check for PII that should not be in training data
• Document the data provenance and version

Step 4: Implement a reproducible training pipeline

• Pin all dependency versions
• Set all random seeds
• Version the dataset (not just the model)
• Store training hyperparameters with model artifacts
• Confirm: can you reproduce this exact model from scratch?

Step 5: Build the evaluation harness before training

Write your evaluation pipeline before training. Evaluations should be:

• Automatic (run in CI)
• Deterministic (same inputs → same scores)
• Multi-dimensional (accuracy, latency, cost, safety, fairness)
• Comprehensive (held-out test set + edge case suite)

Step 6: Train with monitoring

Track: training loss, validation loss, gradient norms. Flag: loss spikes, NaN gradients, overfitting (train loss << val loss), underfitting.