ml-pipeline

ML Pipeline

Purpose

Build a machine learning pipeline that produces the same model twice and behaves in production the way it did in training. The two defining failure modes are irreproducible training and train/serve skew — the model sees different features in production than it saw in training, and quietly degrades.

When to Use

• Building a training or inference pipeline.
• A model that performed well offline and poorly in production.
• Setting up monitoring for a deployed model.
• Establishing a retraining cadence.

Capabilities

• Feature engineering and feature-store design.
• Reproducible training: data versioning, seeds, environment pinning.
• Train/serve consistency.
• Deployment: batch, real-time, shadow.
• Monitoring: data drift, prediction drift, performance decay.

Inputs

• The prediction task and the label definition.
• The data: its sources, its freshness, and its leakage risks.
• Latency and throughput requirements at serving time.

Outputs

• A reproducible training pipeline with a versioned dataset and model.
• Features computed by the same code in training and serving.
• Drift and performance monitoring with alerts.

Workflow

1. Define the label precisely — Including the time at which it becomes known. A label that is only available thirty days after the prediction cannot be used to evaluate a model deployed today, and this constraint shapes everything.
2. Check for leakage first — Any feature computed from information that would not exist at prediction time will make the offline model look excellent and the production model useless. This is the most common and most expensive ML bug.
3. Compute features once, use them twice — The same code path for training and serving. Two implementations of the same feature will diverge, silently, and the model will degrade without any code changing.
4. Version everything — Data, code, environment, hyperparameters, and the model artifact. "Which data produced this model" must be answerable a year later.
5. Shadow before you serve — Run the new model on live traffic, log its predictions, and compare — without acting on them.
6. Monitor drift and performance — Input distribution, prediction distribution, and (when labels arrive) actual accuracy. A model degrades silently; nothing errors.