eval

Eval — AI/LLM Evaluation

Activate When

• User invokes /godmode:eval
• User says "evaluate my model", "benchmark this"
• User says "LLM-as-judge", "compare models", "run evals"
• When deploying or updating any AI/LLM component

Workflow

Step 1: Evaluation Discovery

# Find existing eval infrastructure
find . -name "eval*" -o -name "benchmark*" \
  -o -name "judge*" | grep -v node_modules

# Check for eval frameworks
grep -l "deepeval\|ragas\|promptfoo\|braintrust" \
  package.json pyproject.toml requirements.txt \
  2>/dev/null

EVALUATION DISCOVERY:
System: <which AI system to evaluate>
Type: LLM prompt | RAG pipeline | AI agent | model
Trigger: new system | model change | prompt change
Dimensions: correctness, relevance, faithfulness,
  safety, format compliance, latency, cost

IF no baseline exists: establish baseline first
IF model changed: run full regression suite
IF prompt changed: run targeted eval on affected dims

Step 2: Evaluation Dataset Design

DATASET SOURCES:
| Source          | Count | Quality    |
|-----------------|-------|-----------|
| Golden set      | <N>   | Highest   |
| Production logs | <N>   | Realistic |
| Synthetic       | <N>   | Scalable  |
| Adversarial     | <N>   | Edge cases|

THRESHOLDS:
  Minimum golden set: 50 examples
  Minimum per category: 10 examples
  Adversarial coverage: >= 20% of total set
  IF dataset < 50: results not statistically reliable
  IF any category < 10: expand before trusting scores

Step 3: Framework Selection

| Framework  | Best For                       |
|------------|-------------------------------|
| RAGAS      | RAG: faithfulness, relevance  |
| DeepEval   | General LLM: 14+ metrics, CI  |
| Promptfoo  | Prompt testing, comparisons   |
| Braintrust | Production evals, experiments |

Step 4: LLM-as-Judge

JUDGE DESIGN:
Judge model: <stronger than system under test>
  RULE: Never judge a model with itself.

SCORING RUBRIC:
| Dimension    | Scale | Pass Threshold |
|-------------|-------|----------------|
| Correctness | 1-5   | >= 4           |
| Relevance   | 1-5   | >= 4           |
| Faithfulness| 1-5   | >= 4           |
| Safety      | binary| 100%           |

CALIBRATION:
  Cohen's kappa vs human ratings: >= 0.7 required
  IF kappa < 0.7: refine rubric, add examples
  IF judge disagrees > 30%: re-calibrate