detect-anomalies-aiops

Detect Anomalies for AIOps

See Extended Examples for complete configuration files and templates.

Apply machine learning to detect anomalies in operational metrics, correlate alerts, and reduce false positives.

When to Use

• Operations team overwhelmed by alert volume (>100 alerts/day)
• Need to detect complex multi-metric anomalies (not threshold breaches)
• Seasonal patterns make static thresholds ineffective
• Want to predict issues before they impact users (proactive detection)
• Need to correlate related alerts to identify root cause
• Monitoring system generates too many false positives
• Want to detect subtle performance degradation trends

Inputs

• Required: Time series metrics from monitoring system (CPU, memory, latency, error rate)
• Required: Historical data (30-90 days minimum)
• Optional: Alert history with labels (true positive / false positive)
• Optional: System topology (service dependencies)
• Optional: Log data for correlation
• Optional: Deployment/change events for context

Procedure

Step 1: Set Up Environment and Load Data

Install dependencies and prepare time series data for analysis.

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install anomaly detection libraries
pip install prophet scikit-learn pandas numpy
pip install tensorflow keras  # for LSTM models
pip install pyod  # Python Outlier Detection library
pip install statsmodels  # for statistical methods
pip install prometheus-api-client  # if using Prometheus

# Visualization
pip install plotly matplotlib seaborn

Load and prepare data:

# aiops/data_loader.py
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import List, Dict
import logging

logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)

Got: Time series data loaded with regular intervals, missing values handled, features engineered for ML models.