ml-engineer

ML Engineer

Build and operate production ML systems across the full lifecycle: data → features → training → validation → deployment → monitoring → retraining.

Core Workflow

1. System Analysis

Before building anything:

1. Define the problem — classification, regression, ranking, generation, etc.
2. Assess data — volume, quality, drift patterns, labeling status
3. Set targets — accuracy, latency (<50ms inference), training time (<4h), cost ceiling
4. Map infrastructure — compute (GPU/CPU), storage, orchestration, serving platform
5. Choose deployment strategy — real-time, batch, streaming, edge
6. Plan monitoring — what metrics, what thresholds, who gets paged

2. Pipeline Development

Build modular, versioned pipelines. Each stage should be independently testable and retriable.

Data Validation → Feature Engineering → Training → Validation → Deployment → Monitoring
      ↑                                                                          |
      └──────────────────── Retraining Trigger ←──────────────────────────────────┘

Pipeline principles:

• Data validation FIRST — catch schema drift, missing values, distribution shifts before training
• Version everything: data, features, models, configs, code
• Each stage writes artifacts to a versioned store (MLflow, DVC, W&B)
• Fail fast with clear error messages; never silently produce bad models

See references/pipelines.md for stage-by-stage implementation patterns.

3. Training & Optimization

Select the right approach based on complexity:

Data Size	Complexity	Approach
Small (<10K)	Low	Scikit-learn, XGBoost, single-machine
Medium (10K-1M)	Medium	PyTorch/TF, single GPU, Optuna HPO
Large (1M+)	High	Distributed training (Ray, DeepSpeed), transfer learning
Huge (100M+)	Very high	Multi-node, model sharding, mixed precision

Hyperparameter optimization: Use Optuna (Bayesian) by default. Grid search only for ≤3 params with known ranges. Always set a trial budget and time ceiling.