huawei-cloud-cce-chaos-experiment

Huawei Cloud CCE AZ Power Outage Chaos Experiment

Overview

End-to-end CCE AZ power outage chaos experiment in three phases:

Phase 1: Prepare          Phase 2: Execute           Phase 3: Log Analysis
─────────────             ──────────────             ──────────────────
Discover clusters    →    Pre-check (nodes Ready)  → Load experiment context
Select cluster            Shutdown (BatchStop)       Discover dependencies
Discover node AZs         Monitor (Node/Pod)         Collect logs (LTS + Pod)
Select target AZ          Wait (duration)            Analyze error patterns
Validate compatibility    Rollback (BatchStart)      Generate analysis report
Generate + deploy config  Verify + report

Scenario: Shut down all CCE nodes in one AZ → Kubernetes marks nodes NotReady → Pods evicted and rescheduled to other AZs → start nodes back → verify recovery. Tests cluster HA under AZ-level failure.

Prerequisites

• hcloud CLI — installed and configured with AK/SK (ECS + CCE permissions)
• kubectl — installed and configured for CCE cluster access
• Python 3 — for discovery, validation, execution, and analysis scripts
• Environment variables: HW_ACCESS_KEY, HW_SECRET_KEY, HW_REGION_NAME (optional, defaults to cn-north-4 if unset)

Run environment check:

bash scripts/check_env.sh

If kubectl is missing, check_env.sh auto-installs it. Manual install:

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSL "https://dl.k8s.io/release/v1.28.0/bin/linux/${ARCH}/kubectl" -o /tmp/kubectl_download
sleep 1 && mv /tmp/kubectl_download /usr/local/bin/kubectl && chmod 755 /usr/local/bin/kubectl

Phase 1: Prepare (故障演练准备)

Generates experiment configuration. Does NOT start the experiment.

Step 1.1: Environment Check

bash scripts/check_env.sh

Step 1.2: Determine Region [INTERACTIVE]

Check HW_REGION_NAME environment variable:

• Set → use that region (confirm with user: "Region is , correct?").
• Not set → ask the user which region to use. Do NOT silently default to cn-north-4.