detect-anomalies

Anomaly Detection

Detect anomalies in Axiom datasets by comparing recent patterns to historical baselines using statistical analysis.

Arguments

When invoked with a dataset name (e.g., /detect-anomalies logs), it's available as $ARGUMENTS.

Prerequisites

Statistical anomaly detection requires sufficient data:

• Minimum data points: Z-score and standard deviation need ≥30 samples per bucket for statistical significance
• Historical baseline: At least 24 hours of data for meaningful comparison (methods use 25h lookback)
• Consistent ingestion: Gaps in data collection will skew baselines

If these aren't met, results may be misleading. Consider using simpler threshold-based alerting instead.

Schema Discovery

Always verify field names first:

axiom query "['<dataset>'] | getschema" --start-time -1h

Anomaly Detection Methods

1. Volume Anomaly Detection

Compare recent volume to baseline:

Calculate baseline (past 24h excluding last hour):

axiom query "['<dataset>']
| where _time between (ago(25h) .. ago(1h))
| summarize count() by bin(_time, 1h)
| summarize
    avg_hourly = avg(count_),
    stdev_hourly = stdev(count_)" --start-time -25h -f json

Check recent volume:

axiom query "['<dataset>']
| where _time >= ago(1h)
| summarize
    current_count = count(),
    current_hour = min(_time)" --start-time -1h -f json

Z-score calculation:

• z_score = (current - avg) / stdev
• |z_score| > 2 indicates anomaly

2. New Value Detection

Find values that appeared recently but weren't seen before:

axiom query "['<dataset>']
| where _time >= ago(1h)
| summarize by error_code
| join kind=leftanti (
    ['<dataset>']
    | where _time between (ago(25h) .. ago(1h))
    | summarize by error_code
  ) on error_code" --start-time -25h -f json

Replace error_code with any categorical field (service, endpoint, status).

3. Statistical Outliers

Find values outside normal distribution: