chaos-steady-state-definition

Chaos Steady State Definition

Purpose

Establish a rigorous, measurable definition of "healthy" for every service before chaos experiments run. Without a defined steady state, you cannot determine whether chaos caused harm or whether the system recovered.

---

Prerequisites

• Service definitions from s05 (Service Onboarding)
• Pipeline YAML from s04 (Pipeline Design)
• Prometheus or metrics endpoint accessible
• Observability stack active and collecting metrics
• Performance baselines from s13 (Performance Testing) recommended

Input Contract

Input	Source	Required
Service definitions	s05 (workflow_context.artifacts)	Yes
SLO/SLI targets from PRD	s01 context	Yes
Observability tools available	s20 output or user	Yes
Pipeline CV metrics	s19 output	No
Service dependencies map	s01 context	Yes

Output Contract

Output	Destination	Format
Steady state definition YAML per service	.commandcode/artifacts/steadystate-<service>.yaml	YAML
Prometheus recorded rules	.commandcode/artifacts/prometheus-rules-steadystate.yaml	YAML
LitmusChaos probe configurations	s12 (experiment manifests)	YAML (probes section)
Pre-experiment validation script	.commandcode/artifacts/pre-chaos-check.sh	Bash
Steady state violation alert rules	s21 (alerting)	YAML

---

Steady State Dimensions

Steady State = {Availability} ∩ {Performance} ∩ {Correctness} ∩ {Resources}

Availability:  Is the service responding?        → HTTP probe / TCP probe
Performance:   Is it responding fast enough?     → Latency percentiles
Correctness:   Is it returning correct data?     → Functional probes
Resources:     Is resource use within bounds?    → CPU / memory / connections

---

Step 1 — Define Steady State Metrics

For each service, document the following baseline (measure over 7-day p50):

# steadystate-<SERVICE>.yaml
# Generated by: hybrid-harness-chaos-process-prm
steadyState:
  service: <SERVICE_NAME>
  namespace: <NAMESPACE>
  measuredAt: <DATE>
  environment: <ENV>

  availability:
    httpHealthEndpoint: /health
    expectedStatusCode: 200
    maxResponseTimeMs: 500
    minimumSuccessRate: 99.9    # %

  performance:
    p50LatencyMs: <MEASURED>    # e.g., 45
    p95LatencyMs: <MEASURED>    # e.g., 120
    p99LatencyMs: <MEASURED>    # e.g., 250
    maxAllowedP99Ms: <2x p99>   # Chaos threshold = 2x baseline

  errorRates:
    http5xxRate: <MEASURED>     # e.g., 0.1%
    maxAllowed5xx: 5.0          # % threshold during chaos

  throughput:
    requestsPerSecond: <MEASURED>  # e.g., 1200
    minAllowedRPS: <50% of above>  # e.g., 600 (50% floor)

  resources:
    cpuUsagePercent: <MEASURED>    # e.g., 35%
    memoryUsageMB: <MEASURED>      # e.g., 512MB
    connectionPoolUsage: <MEASURED> # e.g., 40%
    maxCpuAllowed: 80              # % (chaos shouldn't push above this)

  dependencies:
    - name: postgres
      healthCheck: "SELECT 1"
      maxLatencyMs: 50
    - name: redis
      healthCheck: PING
      maxLatencyMs: 10
    - name: payment-gateway
      healthCheck: GET /v1/health
      maxLatencyMs: 200