data-anomaly-detection

Data Anomaly Detection

A skill for identifying anomalies, outliers, and suspicious patterns in research datasets. Combines classical statistical methods with modern machine learning approaches to flag data points that deviate significantly from expected distributions, helping researchers maintain data integrity and uncover genuine scientific findings.

Overview

Anomalous data points in research datasets can arise from measurement errors, instrument malfunction, data entry mistakes, or genuine rare phenomena. Distinguishing between these sources is critical: blindly removing outliers can bias results, while ignoring measurement errors introduces noise. This skill provides a structured framework for detecting, classifying, and handling anomalies in univariate, multivariate, and time-series research data.

The approach follows a three-stage pipeline: detection (flagging candidate anomalies), diagnosis (determining likely cause), and decision (remove, transform, or retain with justification). Every decision is logged for reproducibility and transparent reporting.

Statistical Detection Methods

Univariate Outlier Detection

import numpy as np
from scipy import stats

def detect_univariate_outliers(data: np.ndarray, method: str = 'iqr') -> dict:
    """
    Detect outliers using classical univariate methods.

    Methods:
        'iqr': Interquartile range (1.5x IQR rule)
        'zscore': Z-score threshold (|z| > 3)
        'mad': Median absolute deviation (robust)
        'grubbs': Grubbs' test for single outlier
    """
    results = {'method': method, 'n_total': len(data)}

    if method == 'iqr':
        q1, q3 = np.percentile(data, [25, 75])
        iqr = q3 - q1
        lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
        mask = (data < lower) | (data > upper)

    elif method == 'zscore':
        z = np.abs(stats.zscore(data))
        mask = z > 3

    elif method == 'mad':
        median = np.median(data)
        mad = np.median(np.abs(data - median))
        modified_z = 0.6745 * (data - median) / mad if mad > 0 else np.zeros_like(data)
        mask = np.abs(modified_z) > 3.5

    elif method == 'grubbs':
        # Grubbs' test for the single most extreme value
        n = len(data)
        mean, sd = np.mean(data), np.std(data, ddof=1)
        g = np.max(np.abs(data - mean)) / sd
        t_crit = stats.t.ppf(1 - 0.05 / (2 * n), n - 2)
        g_crit = ((n - 1) / np.sqrt(n)) * np.sqrt(t_crit**2 / (n - 2 + t_crit**2))
        mask = np.abs(data - mean) / sd >= g_crit

    results['outlier_indices'] = np.where(mask)[0].tolist()
    results['n_outliers'] = int(mask.sum())
    results['pct_outliers'] = round(mask.sum() / len(data) * 100, 2)
    return results