gen-evals

You are the great_cto /gen-evals command. You generate synthetic evaluation test cases for a named agent, saving them in the established tests/eval/EVAL-*.md format. This is the Sprint 2 implementation of the self-evolution pattern: synthetic eval dataset generation from agent documentation.

Step 1 — Parse arguments and locate agent

AGENT_NAME="${ARGUMENTS%% *}"  # first word
COUNT=$(echo "$ARGUMENTS" | grep -oE '\-\-count [0-9]+' | grep -oE '[0-9]+' || echo 20)
[ -z "$AGENT_NAME" ] && echo "Usage: /gen-evals <agent-name> [--count N]" && exit 1

# Locate agent file in plugin dir or repo
PLUGIN_DIR=${CLAUDE_PLUGIN_ROOT:-$(ls -d ~/.claude/plugins/cache/*/great_cto/*/ 2>/dev/null | sort -V | tail -1 | sed 's|/$||')}
AGENT_FILE="${PLUGIN_DIR}/agents/${AGENT_NAME}.md"
[ ! -f "$AGENT_FILE" ] && AGENT_FILE="agents/${AGENT_NAME}.md"
[ ! -f "$AGENT_FILE" ] && echo "ERROR: agent not found: ${AGENT_NAME}" && exit 1

echo "Generating $COUNT eval cases for: $AGENT_NAME"
echo "Reading: $AGENT_FILE"

Step 2 — Read agent definition

Read the full agent file. Focus on:

• What the agent is supposed to do (description, step-by-step instructions)
• What it must NOT do (constraints, rejection criteria, anti-patterns)
• What evidence / artifacts it must produce (output format, required fields)
• The domain it covers (archetype, compliance area, specialisation)

This is the source material for generating realistic test cases.

Step 3 — Generate test cases (LLM synthesis)

Based on the agent definition, generate $COUNT realistic, adversarial test cases.

For each test case, produce:

| # | Input / Scenario | Expected behaviour | Pass criterion |

Generation rules (from the PLAN.md pattern):

• Split 60% routine / 40% adversarial (edge cases, subtle failure modes)
• Routine: valid inputs where the agent should succeed clearly
• Adversarial: inputs designed to expose the most common failure modes for this agent type
• Expected behaviour: rubric-based, not exact-match (e.g. "flags at least 2 injection vectors" not "outputs the word SQL")
• Pass criterion: objectively verifiable — a judge model can determine pass/fail