huawei-cloud-modelarts-training-diagnosis

ModelArts Training Job Fault Diagnosis

Overview

This skill provides automated fault diagnosis for Huawei Cloud ModelArts training jobs. It calls ModelArts log and event APIs via hcloud CLI to collect runtime information, analyzes training job failures/timeouts/stuck jobs, and outputs diagnosis conclusions with fix suggestions and confidence levels.

Architecture

User Input
    ↓
Phase 1: Task Discovery (ListTrainingJobs)
    ↓
Phase 2: Status Assessment (ShowTrainingJobDetails)
    ↓
[Early Exit if traceback found in status.task_statuses[].message]
    ↓ (if no traceback)
Phase 3: Information Collection
    ├─ Main Path: ListTrainingJobEvents, ListTrainingJobStages
    └─ Extended Path: ShowTrainingJobLogsPreview, ShowObsUrlOfTrainingJobLogs
    ↓
Phase 4: Analysis (confidence-based inference)
    ↓
Phase 5: Output (diagnosis report + fix suggestions)

Applicable Scenarios

• Training job failure (status.phase = "Failed")
• Training job timeout (status.phase = "Timeout")
• Training job abnormal (status.phase = "Abnormal")
• Training job stuck (running long time with no progress)
• Resource shortage causing training failure

Typical Use Cases

• "My training job failed, help me diagnose"
• "Training job is stuck, no progress for hours"
• "Training job timeout, what went wrong?"
• "Scan all failed training jobs in my account"
• "Training job error code 1.015, what does it mean?"

Prerequisites

hcloud CLI Installation

• Version: 7.2.2 or higher
• Verification: hcloud version should return version >= 7.2.2
• Installation guide: See references/cli-installation-guide.md

Authentication Configuration

• AK/SK configured in ~/.hcloud/config.json
• Default region: cn-north-4
• project_id configured in profile
• skipSecureVerify=true (for WSL environment)

Verification command:

hcloud configure list

IAM Permissions

This skill requires read-only permissions for ModelArts training APIs.