evals-clarify

evals-clarify

What this skill does

Conducts axial coding following EDD Principles III & IX to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.

Output:

1. Clustered Criteria - Related patterns grouped into coherent evaluation themes
2. Adversarial Examples - Generated attack scenarios and edge cases for robustness
3. Published Goldset - Accepted criteria in evals/{system}/goldset.md with full documentation
4. Holdout Dataset - Reserved test set (20%) for unbiased evaluation validation
5. JSON Configuration - Auto-generated goldset.json for system consumption
6. Auto-handoff to /evals-implement for grader generation

Key EDD Principles Applied:

• Principle III: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
• Principle IX: Test Data as Code - Adversarial generation, holdout splits, version control
• Principle II: Binary Pass/Fail - Maintain strict binary evaluation throughout
• Principle I: Spec-Driven Contracts - Criteria validate spec compliance

When to use

• After /evals-specify: Refine and accept draft criteria into goldset
• Dataset maintenance: Balance pass/fail examples or add adversarial cases
• Adding holdout split: Isolate validation data from training data

When NOT to use

• No draft criteria exist: Run /evals-specify to discover patterns first
• Grader generation: Use /evals-implement to convert accepted goldset into code

Process

User Input

$ARGUMENTS

• --accept IDS — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
• --merge IDS — Merge related criteria (e.g., "EVAL-001+EVAL-002")
• --split ID — Split complex criterion into multiple focused criteria
• --holdout-ratio RATIO — Holdout percentage (default: 0.2, range: 0.1-0.3)

Execution Steps

Phase 1: Axial Coding & Clustering

• Group related draft patterns into coherent themes.
• Resolve any overlaps or duplicate criteria.