data-drift-strategy

Data Drift Strategy

Purpose: Detect and manage changes in input data distributions that degrade GenAI agent quality and ML pipeline reliability. Covers LLM input monitoring, embedding drift, RAG retrieval degradation, and classical feature distribution tracking.

---

When to Use This Skill

• Monitoring LLM input patterns in production (query length, topic distribution, language mix)
• Detecting embedding drift (vector space distribution shifts in RAG systems)
• Tracking RAG retrieval quality degradation (relevance scores, retrieval hit rates)
• Identifying user intent drift (new topics, out-of-scope queries, adversarial inputs)
• Monitoring input feature distributions in production ML systems
• Building data quality validation gates in ETL/ML pipelines
• Detecting schema drift (new columns, type changes, missing fields)
• Designing retraining triggers based on data distribution shifts
• Establishing data governance policies for model training data

Prerequisites

• Reference dataset (baseline distribution from training data or initial deployment)
• Data pipeline with logging/profiling capabilities
• For GenAI: Input logging enabled on the agent (query text, timestamp, response, latency)
• Statistical testing library (scipy, evidently, or equivalent)

Decision Tree

Data drift concern?
+- GenAI / LLM application?
|  +- Query patterns shifting?
|  |  +- Length distribution changed? -> Update test dataset, adjust token budgets
|  |  +- New topic clusters emerging? -> Expand scope or add guardrails
|  |  +- Language mix changed? -> Add multilingual testing
|  +- Embedding drift detected?
|  |  +- Cosine similarity distribution shifted? -> Re-index or fine-tune embeddings
|  |  +- New clusters forming? -> Knowledge base gaps, update content
|  |  +- Retrieval scores dropping? -> RAG pipeline degradation
|  +- Adversarial patterns increasing?
|  |  +- Prompt injection attempts? -> Strengthen guardrails, log patterns
|  |  +- Jailbreak frequency rising? -> Update safety filters
|  +- Conversation patterns changing?
|     +- Turn depth increasing? -> Agent may be struggling, check quality
|     +- Tool usage shifting? -> Review tool definitions, add new tools
|     +- Satisfaction signals declining? -> Full quality audit needed
+- Traditional ML model?
|  +- Schema changed?
|  |  +- New columns added? -> Schema evolution (validate compatibility)
|  |  +- Columns removed? -> Breaking change (alert immediately)
|  |  +- Type changed? -> Data pipeline bug (investigate source)
|  +- Feature distribution shifted?
|  |  +- Single feature? -> Upstream data source change
|  |  +- Multiple features? -> Systemic shift (new data segment or pipeline change)
|  |  +- Correlations changed? -> Relationship drift (may affect model assumptions)
|  +- Data quality degraded?
|     +- Missing values increased? -> Source system issue
|     +- Outliers increased? -> Validation rules needed
|     +- Duplicates increased? -> Deduplication pipeline issue
+- No visible issues?
   +- Set up proactive profiling -> Baseline all features
   +- Schedule periodic checks -> Compare current vs. reference