ML-MLOps-Portfolio: Skill for Claude Code

Model Retraining

Authorization Protocol

This skill spans three authorization layers, aligned with the Agent Behavior Protocol (AGENTS.md):

Phase	Mode	What happens
Training (MLflow run)	AUTO	Agent may run training, log to MLflow, produce artifacts
Transition to Staging	CONSULT	Agent presents metrics + quality gates + drift diff; human approves via MLflow UI or /promote-model PR
Transition to Production	STOP	Never transitions directly. Opens PR, waits for Tech Lead approval via GitHub Environment production

Automatic STOP escalation

Even in AUTO phase, escalate to STOP if the new model exhibits any of:

• Primary metric > 0.99 without explanation (D-06 — investigate leakage)
• Fairness DIR in [0.80, 0.85] (marginal — human judgment required)
• Metric regression > 5% vs current production
• Any quality gate fails

Emit structured signal:

[AGENT MODE: STOP]
Operation: Model retraining for {service}
Reason: Fairness DIR = 0.82 (marginal, requires human review)
Waiting for: Engineer inspection + either ADR documenting decision OR retraining with fairness-aware loss

Step 1: Validate Retraining Trigger

Before retraining, confirm the trigger:

• Drift alert: PSI ≥ threshold on critical feature (check Prometheus/Grafana)
• Metric degradation: Rolling metric below quality gate (check monitoring)
• Scheduled: Periodic retraining per policy
• Manual: Engineer-initiated (document reason)

Step 2: Download Fresh Data

# Download latest production data
gsutil cp gs://{data-bucket}/{service}/production_data_latest.csv data/raw/
# Or from AWS:
aws s3 cp s3://{data-bucket}/{service}/production_data_latest.csv data/raw/

Step 3: Validate Data Before Training

python -c "
from src.{service}.schemas import ServiceInputSchema
import pandas as pd
import pandera as pa

df = pd.read_csv('data/raw/production_data_latest.csv')
ServiceInputSchema.validate(df)
print(f'Validation passed: {len(df)} rows, {len(df.columns)} columns')
"