root-cause-analysis

Root Cause Analysis

Investigate failed jobs by correlating Ansible Automation Platform (AAP) job logs with Splunk OCP pod logs and analyzing AgnosticD/AgnosticV configuration to identify root causes.

Automatic Execution

When a user asks to analyze a failed job, execute these steps automatically.

The skill's base path is provided when this skill is invoked. Run scripts relative to this folder.

Setup (run once per session if .venv doesn't exist)

# Create virtual environment and install dependencies
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt

Step 1-3: Run the analysis CLI

# Option 1: By job ID (searches JOB_LOGS_DIR automatically)
.venv/bin/python scripts/cli.py analyze --job-id <JOB_ID>

# Option 2: By explicit path
.venv/bin/python scripts/cli.py analyze --job-log <path-to-job-log>

The skill automatically searches for job logs in the configured JOB_LOGS_DIR (set in .env).

Step 4: Read outputs and analyze with AgnosticD/V context

After running the CLI, read the generated files from the .analysis/ folder in this skill directory:

.analysis/<job-id>/step1_job_context.json
.analysis/<job-id>/step2_splunk_logs.json
.analysis/<job-id>/step3_correlation.json

Then perform enhanced analysis:

1. Parse job metadata to identify platform/demo/env from job_name pattern
2. Fetch AgnosticV configuration hierarchy from rhpds/agnosticv
3. Fetch AgnosticD workload code from redhat-cop/agnosticd
4. Apply investigation rules to identify root cause
5. Generate summary with actionable recommendations

Provide a summary to the user with:

1. Job Details: ID, status, GUID, namespace, platform, demo
2. Failed Task(s): IMPORTANT - Preserve ALL fields from step1 failed_tasks:
   • Task: task name
   • Play: play name
   • Role: role name
   • Task Action: task_action (Ansible module)
   • Error: error_message
   • Task Duration: duration seconds
   • Location: Both original task_path AND derived GitHub path
     • Original: /home/runner/.ansible/collections/... (from task_path field)
     • GitHub: repository:file_path:line (parsed from original)
     • Example: agnosticd/core_workloads:roles/ocp4_workload_gitops_bootstrap/tasks/workload.yml:74
3. Configuration Context: Relevant variables, missing vars, secrets references
4. Correlation: How AAP logs link to Splunk pod logs (GUID, namespace, time overlap)
5. Correlated Pods: Pods found in Splunk during the job window
6. Root Cause: Analysis of why the job failed (configuration vs. infrastructure vs. workload bug)
7. Recommendations: Specific file changes with paths and suggested values

---