ml-pipeline

Machine Learning Pipeline - Multi-Agent MLOps Orchestration

Design and implement a complete ML pipeline for: $ARGUMENTS

Thinking

This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:

• Phase-based coordination: Each phase builds upon previous outputs, with clear handoffs between agents
• Modern tooling integration: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving
• Production-first mindset: Every component designed for scale, monitoring, and reliability
• Reproducibility: Version control for data, models, and infrastructure
• Continuous improvement: Automated retraining, A/B testing, and drift detection

The multi-agent approach ensures each aspect is handled by domain experts:

• Data engineers handle ingestion and quality
• Data scientists design features and experiments
• ML engineers implement training pipelines
• MLOps engineers handle production deployment
• Observability engineers ensure monitoring

Phase 1: Data & Requirements Analysis

Deliverables:

1. Data source audit and ingestion strategy:

   • Source systems and connection patterns
   • Schema validation using Pydantic/Great Expectations
   • Data versioning with DVC or lakeFS
   • Incremental loading and CDC strategies

2. Data quality framework:

   • Profiling and statistics generation
   • Anomaly detection rules
   • Data lineage tracking
   • Quality gates and SLAs

3. Storage architecture:

   • Raw/processed/feature layers
   • Partitioning strategy
   • Retention policies
   • Cost optimization

Provide implementation code for critical components and integration patterns.