openakari: Skill for Claude Code

/review [metrics | findings]

Validate experiment outputs in two modes. Metrics mode checks whether metric computations are meaningful given the experimental setup (run before writing findings). Findings mode checks whether written conclusions are valid (run after writing findings). If no mode is specified, run both in sequence: metrics first, then findings.

When to use this vs alternatives

• Use /review metrics when you have metric definitions or computed values and want to check whether they are meaningful given the experimental setup.
• Use /review findings when findings have been written and you want to validate each claim.
• Use /review (no mode) to run the full pipeline: metrics validation → findings validation.
• Use /critique for a broad adversarial review across 9 failure dimensions. Critique is wider but shallower; /review goes deeper on metric validity and finding correctness.
• Use /diagnose when you want to understand what results mean — error patterns, root causes, hypotheses. Diagnose interprets results; /review checks whether they are interpretable and correctly stated.

---

Metrics mode

1. Extract the constraint set

Identify the fixed parameters of the experiment:

• Response schema: What values can the model output? (e.g., Literal["A", "B"], 1-5 scale, free text)
• n_runs: How many repeated calls per evaluation?
• Temperature: Is there randomness across runs?
• Sample size: How many items/pairs/tasks?
• Ground truth structure: Does ground truth include ties, ordinal rankings, continuous scores?
• Aggregation method: How are repeated runs combined? (majority vote, mean, threshold)

2. For each metric, apply these tests

Degeneracy test

Given the constraints, can this metric take more than one value? If the setup forces the metric to a constant regardless of model behavior, it is degenerate.

Examples: accuracy with n_runs=1 → always (non-tie GTs / total); confidence range with binary schema → always 0% or 100%; inter-run agreement with temperature=0 → always ~100%.