chaos-observability-integration

Chaos Observability Integration (s20)

Purpose

Establish comprehensive observability for chaos experiments — ensuring every fault is visible, every probe is connected to real metrics, and every team member can see what's happening in real-time during chaos runs.

---

Prerequisites

• Continuous verification config from s21 (CV Verification)
• Chaos experiment results from s14-s19
• Prometheus and Grafana deployed and accessible
• Service metrics endpoints available
• Dashboard creation permissions in Grafana

Input Contract

Input	Source	Required
Steady state metrics definitions	s15 output	Yes
Experiment manifests (probe configs)	s12-s17 outputs	Yes
Observability tool preferences	s02 taste (observability)	Yes
Alert routing preferences	s02 taste or s01	Yes
CV verification config	s19 output	No

Output Contract

Output	Destination	Format
Chaos Grafana dashboard JSON	.commandcode/artifacts/dashboard-chaos.json	JSON
Prometheus alert rules for chaos	.commandcode/artifacts/alerts-chaos.yaml	YAML
Observability health check script	.commandcode/artifacts/obs-check.sh	Bash
Probe validation report	s18 (game day)	Markdown
Chaos metrics feed	s21 (alerting), s24 (scoring)	Prometheus queries

---

Chaos Observability Stack

Experiment Runner (LitmusChaos/HCE)
    │  Exposes chaos metrics on :8080/metrics
    ▼
Prometheus (scrapes every 15s)
    │
    ├── Grafana Dashboard (real-time chaos view)
    ├── AlertManager (chaos-specific alert rules)
    └── S24 Scoring Engine (resilience calculation)

---

Step 1 — Chaos Grafana Dashboard

{
  "dashboard": {
    "title": "Chaos Engineering — <SERVICE>",
    "uid": "chaos-<service>",
    "tags": ["chaos", "resilience", "<service>"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Chaos Experiment Status",
        "type": "stat",
        "targets": [{
          "expr": "litmuschaos_experiment_verdict{app='<SERVICE>'}",
          "legendFormat": "{{experiment_name}}"
        }],
        "fieldConfig": {
          "defaults": {
            "mappings": [
              {"type": "value", "value": "0", "text": "Passed"},
              {"type": "value", "value": "1", "text": "Failed"}
            ],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {"color": "green", "value": null},
                {"color": "red", "value": 1}
              ]
            }
          }
        }
      },
      {
        "title": "Resilience Score (Real-time)",
        "type": "gauge",
        "targets": [{
          "expr": "litmuschaos_resilience_score{app='<SERVICE>'}"
        }],
        "fieldConfig": {
          "defaults": {
            "min": 0, "max": 100,
            "thresholds": {
              "steps": [
                {"color": "red", "value": null},
                {"color": "yellow", "value": 60},
                {"color": "green", "value": 80}
              ]
            }
          }
        }
      },
      {
        "title": "Error Rate During Chaos",
        "type": "timeseries",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{namespace='<NAMESPACE>',status=~'5..'}[1m])) / sum(rate(http_requests_total{namespace='<NAMESPACE>'}[1m])) * 100",
            "legendFormat": "Error Rate %"
          },
          {
            "expr": "5",
            "legendFormat": "Threshold (5%)"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "thresholds": {
              "steps": [
                {"color": "green", "value": null},
                {"color": "red", "value": 5}
              ]
            }
          }
        }
      },
      {
        "title": "P99 Latency During Chaos",
        "type": "timeseries",
        "targets": [{
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_ms_bucket{namespace='<NAMESPACE>'}[1m])) by (le))",
          "legendFormat": "P99 (ms)"
        }]
      },
      {
        "title": "Pod Status During Chaos",
        "type": "stat",
        "targets": [
          {
            "expr": "count(kube_pod_status_ready{namespace='<NAMESPACE>', condition='true'})",
            "legendFormat": "Ready"
          },
          {
            "expr": "count(kube_pod_status_ready{namespace='<NAMESPACE>', condition='false'})",
            "legendFormat": "Not Ready"
          }
        ]
      },
      {
        "title": "Active Faults",
        "type": "table",
        "targets": [{
          "expr": "litmuschaos_active_experiments{namespace='<NAMESPACE>'}",
          "format": "table"
        }]
      },
      {
        "title": "Probe Status",
        "type": "status-history",
        "targets": [{
          "expr": "litmuschaos_probe_success_total{namespace='<NAMESPACE>'}",
          "legendFormat": "{{probe_name}}"
        }]
      }
    ]
  }
}