simulation-failure-triage

Simulation Failure Triage

Goal

Classify common simulation failure signatures and return immediate actions, retry ladders, and stop conditions.

Requirements

• Python 3.10+
• No external dependencies
• Works on Linux, macOS, and Windows

Inputs to Gather

Input	Description	Example
Code	Simulation code	LAMMPS, VASP, MOOSE, QE
Stage	Setup, runtime, postprocess	runtime
Symptoms	Failure signs	nan,pressure-blowup
Log text or file	Error evidence	Lost atoms, ZBRENT
Recent change	Last modified setting	larger timestep

Decision Guidance

• First preserve evidence: logs, inputs, executable version, and scheduler output.
• Separate setup errors from numerical instability and physical model issues.
• Retry with a single controlled change.
• Stop retrying when the result becomes scientifically meaningless or a required model input is missing.

Script Outputs

scripts/failure_triage.py emits:

• likely_causes
• immediate_actions
• retry_ladder
• stop_conditions
• evidence

Workflow

python3 skills/robustness/simulation-failure-triage/scripts/failure_triage.py \
  --code LAMMPS \
  --stage runtime \
  --symptoms nan,pressure-blowup \
  --recent-change "increased timestep" \
  --json

Error Handling

Invalid stages or oversized log files stop with exit code 2. Unknown symptoms are retained as custom evidence.

Limitations

This skill gives first-response triage. It does not guarantee that a failed simulation can be repaired.

Verification checklist

• Recorded the preserved-evidence set BEFORE any rerun: copied logs, input decks, scheduler output, and the executable/version string (per the first immediate_actions item) so the original failure is reproducible.
• Quoted the FIRST warning/error from the run, not just the final crash line, and confirmed evidence.log_excerpt (first 500 chars, original casing) captures it; re-run with --log-file/--log-text if the excerpt misses the root signature.
• Logged every likely_causes entry with its symptom, category, and first_action, and verified no real signature landed in the category: custom bucket because its keyword was absent from LOG_HINTS/PATTERNS.
• Confirmed the failure category is correct rather than mislabeled: a segfault/SIGSEGV/signal 11/core-dumped maps to crash (not corrupted-output), and an OOM/bad_alloc/oom-kill maps to out-of-memory (not incomplete-run).
• Applied the retry_ladder strictly one change at a time, recording the single parameter changed and its effect at each rung (including the memory rung when memory-bound).
• Checked every stop_conditions entry and halted the retry loop instead of stacking arbitrary stabilizing tweaks when any condition is met (e.g. unverifiable potential/pseudopotential, results dependent on arbitrary stabilizers).
• After any numerical recovery, handed off to the simulation-validator skill for conservation / physical-bounds / "can I trust these results" checks rather than declaring success here.