root-cause-analysis

Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.

Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically. The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

Setup (run once per session if .venv doesn't exist)

# Create virtual environment and install dependencies
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt

Step 1-3: Run the analysis CLI

# Option 1: By job ID (searches JOB_LOGS_DIR automatically)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID>

# Option 2: By explicit path
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>

The skill automatically searches for job logs in the configured JOB_LOGS_DIR (set in .env).

Step 4: Fetch GitHub Data (Automated)

# Fetch all GitHub files automatically via GitHub API
.venv/bin/python scripts/step4_fetch_github.py --job-id <JOB_ID>

This script automatically parses job metadata, fetches AgnosticV configs and AgnosticD workload code, and reports success/404 status.

Output: .analysis/<job-id>/step4_github_fetch_history.json

MANDATORY: GitHub MCP Tools for 404 Verification:

• ALWAYS check 404 files via MCP server when step4 reports "error": "all_paths_failed" or any "status": "404" in paths_tried
• DO NOT skip 404 verification - the script may have incorrect path parsing
• Process:
  1. Check parent directory first using mcp__github__get_file_contents to list actual folder/file names (reveals case sensitivity, hyphens vs underscores)
  2. If parent check fails, use mcp__github__search_code to locate files by filename or partial path
  3. Document findings: wrong format → parser bug, missing → truly missing, empty → rare
  4. No assumptions: "empty", "test", "demo" in names are just labels - always verify 404s