prometheus-analysis

Prometheus Analysis

Overview

Prometheus is a time-series metrics collection and alerting system widely used for monitoring production systems. This skill provides guidance for querying Prometheus metrics, interpreting alert data, and using metrics for root cause analysis.

When to Use This Skill

Apply this skill when:

• Analyzing Prometheus alerts that have fired
• Querying metrics to understand system behavior
• Investigating metric anomalies or spikes
• Correlating metrics with incidents
• Writing PromQL queries for specific metrics
• Interpreting time-series data patterns

Prometheus Fundamentals

Metric Types

Counter: Cumulative value that only increases (e.g., total requests, error count)

• Use rate() or increase() to get per-second rate or total increase
• Example: http_requests_total

Gauge: Value that can go up or down (e.g., CPU usage, memory usage, queue depth)

• Query directly or use functions like avg_over_time()
• Example: node_memory_usage_bytes

Histogram: Distribution of values in buckets (e.g., request durations)

• Provides _sum, _count, and _bucket metrics
• Use for percentile calculations
• Example: http_request_duration_seconds

Summary: Similar to histogram but with pre-calculated quantiles

• Example: http_request_duration_seconds{quantile="0.95"}

Time Series Format

Metrics have format: metric_name{label1="value1", label2="value2"}

Example: http_requests_total{method="POST", status="500", service="api"}

Querying Prometheus

Basic Queries

Instant query (current value):

http_requests_total

Range query (over time):

http_requests_total[5m]

Filter by labels:

http_requests_total{status="500", service="api"}

Rate of increase (per-second rate):

rate(http_requests_total[5m])

Common Aggregation Functions

Sum across dimensions:

sum(rate(http_requests_total[5m])) by (status)

Average:

avg(node_memory_usage_bytes) by (instance)