wshobson-ml-pipeline-workflow

---

name: ml-pipeline-workflow description: Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.

ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill

• Building new ML pipelines from scratch
• Designing workflow orchestration for ML systems
• Implementing data → model → deployment automation
• Setting up reproducible training workflows
• Creating DAG-based ML orchestration
• Integrating ML components into production systems

What This Skill Provides

Core Capabilities

1. Pipeline Architecture

   • End-to-end workflow design
   • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
   • Component dependencies and data flow
   • Error handling and retry strategies

2. Data Preparation

   • Data validation and quality checks
   • Feature engineering pipelines
   • Data versioning and lineage
   • Train/validation/test splitting strategies

3. Model Training

   • Training job orchestration
   • Hyperparameter management
   • Experiment tracking integration
   • Distributed training patterns

4. Model Validation

   • Validation frameworks and metrics
   • A/B testing infrastructure
   • Performance regression detection
   • Model comparison workflows

5. Deployment Automation

   • Model serving patterns
   • Canary deployments
   • Blue-green deployment strategies
   • Rollback mechanisms

Reference Documentation