ml-best-practices

ML Best Practices

Model Selection Guidelines

Problem Type Classification

• Supervised Learning: Labeled data for training
  • Regression: Predict continuous values (Linear Regression, Random Forest, Gradient Boosting)
  • Classification: Predict discrete labels (Logistic Regression, SVM, Decision Trees, Neural Networks)
• Unsupervised Learning: Unlabeled data exploration
  • Clustering: Group similar data points (K-Means, DBSCAN, Hierarchical)
  • Dimensionality Reduction: Reduce feature space (PCA, t-SNE, UMAP)
  • Anomaly Detection: Identify outliers (Isolation Forest, One-Class SVM)
• Reinforcement Learning: Learn through interaction with environment
  • Policy-based: Learn policy directly (REINFORCE, PPO)
  • Value-based: Learn value function (DQN, SARSA)

Algorithm Selection Criteria

• Data Size: Small vs. large datasets
• Feature Types: Numerical, categorical, text, image
• Interpretability: Need for model explanations
• Training Time: Constraints on model training
• Inference Latency: Real-time vs. batch predictions
• Accuracy Requirements: Trade-offs with complexity

Common ML Frameworks

• scikit-learn: Traditional ML algorithms, easy to use
• TensorFlow/Keras: Deep learning, production-ready
• PyTorch: Research-friendly, dynamic computation graphs
• XGBoost/LightGBM: Gradient boosting for tabular data
• Hugging Face Transformers: Pre-trained NLP models

Feature Engineering Techniques

Numerical Features

• Scaling: Standardization (z-score) or Min-Max scaling
• Binning: Convert continuous to categorical
• Polynomial Features: Create interaction terms
• Log Transformations: Handle skewed distributions
• Normalization: Scale to unit norm

Categorical Features

• One-Hot Encoding: Binary columns for each category
• Label Encoding: Map categories to integers
• Ordinal Encoding: Preserve order for ordinal categories
• Target Encoding: Replace with target mean (with regularization)
• Embedding: Learn dense representations (for high cardinality)