model-drift-management

Model Drift Management

Purpose: Detect model performance degradation in GenAI agents and traditional ML systems. Covers LLM output quality monitoring, prompt regression detection, model version change management, and classical statistical drift detection.

When to Use This Skill

• Monitoring LLM output quality in production (hallucination rate, format compliance, coherence)
• Detecting prompt regression after prompt edits or model version changes
• Managing model version transitions (provider silent updates, planned migrations)
• Implementing LLM-as-judge evaluation pipelines for continuous quality monitoring
• Monitoring model performance in production (accuracy decay, prediction shifts)
• Implementing drift detection pipelines (concept drift, prior probability shift)
• Designing change management workflows for model retraining and replacement
• Building model governance and versioning policies
• Setting up alerting thresholds for model degradation

Prerequisites

• A deployed GenAI agent or ML model with inference logging
• OpenTelemetry tracing enabled (for GenAI) or metric pipeline (for ML)
• Evaluation baseline saved from last known-good model version
• Access to ground truth labels or LLM-as-judge evaluator (for quality scoring)

Decision Tree

Model in production?
+- GenAI / LLM agent?
|  +- Output quality declining? -> LLM Drift (see GenAI Drift section below)
|  +- Provider updated the model silently? -> Model version drift (pin versions)
|  +- Prompt was edited? -> Prompt regression (run eval baseline comparison)
|  +- Structured output breaking? -> Format compliance drift
|  +- Tool calls failing more often? -> Tool accuracy drift
|  +- Latency or cost spiking? -> Operational drift (check token usage)
+- Traditional ML model?
|  +- Performance degrading?
|  |  +- Sudden drop? -> Concept drift (data distribution changed)
|  |  +- Gradual decline? -> Model staleness (retrain on recent data)
|  |  +- Intermittent? -> Check data pipeline quality first
|  +- Predictions shifting?
|  |  +- Output distribution changed? -> Prior probability shift
|  |  +- Confidence scores dropping? -> Feature drift (inputs changing)
|  |  +- New unseen categories? -> Covariate shift (retrain or extend)
+- No visible issues?
   +- Set up proactive monitoring -> Evaluation baselines + statistical tests
   +- Schedule periodic evaluation -> Compare current vs. baseline metrics