log-aggregation

Skill: Log Aggregation

Expertise: Loki (Grafana stack), Promtail/Fluent Bit, structured JSON logging, log-based alerting, ELK basics.

When to load

When setting up log collection, writing log queries, debugging missing logs, or adding log-based alerts.

Loki Stack (K8s — recommended)

# Promtail DaemonSet auto-discovers K8s pod logs
# Install via helm:
helm upgrade --install loki grafana/loki-stack \
  -n monitoring \
  -f loki-values.yaml

# loki-values.yaml
loki:
  auth_enabled: false
  limits_config:
    retention_period: 720h   # 30 days
    ingestion_rate_mb: 16
    max_streams_per_user: 10000
  storage_config:
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-index
    filesystem:
      directory: /data/loki/chunks

promtail:
  config:
    clients:
      - url: http://loki:3100/loki/api/v1/push
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          - docker: {}     # parse Docker JSON log format
          - json:          # extract fields from app JSON logs
              expressions:
                level: level
                trace_id: trace_id
                service: service
          - labels:
              level:
              service:

LogQL Queries

# All error logs from a service in last 5 min
{namespace="production", app="order-service"} |= "ERROR"

# Parse JSON and filter by field
{namespace="production"} | json | level="error" | trace_id != ""

# Count errors per service (for alerting)
sum by (service) (
  count_over_time({namespace="production"} | json | level="error" [5m])
)

# Log rate (to detect log explosion)
sum(rate({namespace="production"}[5m])) by (app)

# Find slow requests from logs
{app="api-gateway"} | json | response_time_ms > 500

Structured Logging Standards

# Python — structlog
import structlog

log = structlog.get_logger()

# Always include: service, version, trace_id, span_id, level
log.info("order.created",
    order_id="ord-123",
    user_id="usr-456",    # OK in log; NOT in metrics labels
    amount_cents=4999,
    # trace_id injected automatically via TraceContextFilter
)

# Output (JSON):
# {"event": "order.created", "level": "info", "order_id": "ord-123",
#  "trace_id": "abc123def456", "span_id": "789xyz", "timestamp": "..."}