reanalyze

Skill: reanalyze

Re-run analyze-run.sh on every run matching an RQ selector, reaggregate via aggregate-by-query.py, and propose findings updates. No new runs are started — this skill only refreshes existing data.

Typical use cases:

• Pipeline fix (analyze-run.sh changed, verification adapter added)
• Manual run cleanup (crash runs deleted, runs re-added)
• ESLint/SonarJS config change affecting smell metrics
• New fields added to metrics.json

Argument

• RQ-N (e.g. RQ-prompt-known-kata) or a direct path to an RQ dir.
• If not given: ask back ("Which RQ? e.g. RQ-prompt-known-kata").
• RQ dirs live in four subtrees: research/questions-claude/<chapter>-*/, research/questions-opencode/<chapter>-*/, research/questions-cross/<chapter>-*/, and research/workflow-dev/<chapter>-*/. The chapter prefix is an ordering label; the stable id is the frontmatter id: (a slug like RQ-prompt-known-kata). Resolve it to a path by exact-line id-grep across all subtrees:

  RQ_DIR=$(grep -rlE "^id:[[:space:]]*RQ-prompt-known-kata[[:space:]]*$" \
             research/questions-claude/*/README.md \
             research/questions-opencode/*/README.md \
             research/questions-cross/*/README.md \
             research/workflow-dev/*/README.md \
           2>/dev/null | head -1 | xargs -r dirname)
  

Phases

Run sequentially. On errors in any phase, stop and ask the user, do not skip ahead.

---

Phase 1 — Resolve & Match

1. Resolve the RQ path into $RQ_DIR via the id-grep in "Argument" above. On no match, ask the user; on multiple, take the first and inform.
2. Read $RQ_DIR/README.md — parse frontmatter for controls and factors (needed to build the selector).
3. Find all matching runs in experiments/runs/:

   for d in experiments/runs/*/; do
       jq -e 'select(
           .kata == "<expected>" and
           .workflow == "<expected>" and
           (.model | test("<model-pattern>"))
       )' "$d/metrics.json" > /dev/null 2>&1 && echo "$d"
   done
   

   The selector logic mirrors aggregate-by-query.py: kata = kata_base + "-" + prompt for each prompt value (from factors or controls), model from factors or controls, workflow from controls. Match against .kata, .workflow, .model in each run's metrics.json.
4. Report: "Found N runs matching RQ-X selector."
5. If N == 0: STOP — "No matching runs found. Check that experiments/runs/ contains runs with the expected kata/workflow/model."