domino-model-monitoring

Domino Model Monitoring Skill

Description

This skill helps users monitor deployed models in Domino, including drift detection, model quality tracking, and alerting.

Activation

Activate this skill when users want to:

• Monitor deployed model performance
• Set up drift detection
• Configure monitoring alerts
• Analyze prediction data
• Understand model degradation

What is Model Monitoring?

Domino Model Monitoring provides:

• Data Drift Detection: Detect changes in input data distributions
• Model Quality Tracking: Monitor prediction accuracy over time
• Alerting: Get notified when metrics exceed thresholds
• Prediction Capture: Log predictions for analysis
• Reproducibility: Diagnose issues with captured data

Setting Up Monitoring

Prerequisites

1. Deployed Model API in Domino
2. Training dataset (for baseline)
3. Ground truth data (optional, for quality metrics)

Enable Monitoring

1. Go to your Model API page
2. Click Monitoring tab
3. Click Set Up Monitoring
4. Upload training dataset
5. Configure drift detection settings

Register Training Data

# Training data provides baseline for drift detection
# Upload via UI or programmatically

import pandas as pd

# Your training data
train_df = pd.read_csv("training_data.csv")

# Save for monitoring setup
train_df.to_csv("/mnt/artifacts/training_data.csv", index=False)

Drift Detection

Types of Drift

Drift Type	Description
Data Drift	Input feature distributions change
Concept Drift	Relationship between inputs and outputs changes
Prediction Drift	Output distribution changes

Statistical Tests

Domino supports multiple drift detection tests:

Test	Best For
Kullback-Leibler Divergence	General-purpose, most common
Population Stability Index (PSI)	Finance industry standard
Wasserstein Distance	Comparing distributions
Energy Distance	Multivariate distributions