evaluation-protocol-design

Evaluation Protocol Design (Sub-skill of Method Framework Design)

Overview

Applies to Type M, Type C, and Type H projects. The evaluation protocol is the contract between you and the scientific community. Once locked, it defines exactly how success is measured. Designing it after seeing results is not design — it is rationalization.

Type routing: Type M and Type H follow the standard protocol below. Type C follows the Type C Protocol section at the end of this skill.

Step-by-Step Protocol Design

For each item below, present the recommendation to the user as a multiple-choice question where possible. Justify every recommendation. Do NOT proceed to the next item until the user confirms the current one.

1. Select Primary Metrics

Propose 1–3 primary metrics. For each:

• What it measures (e.g., "F1 measures balanced precision-recall on imbalanced classes")
• Why it is appropriate for this task and venue
• Known limitations (no metric is perfect — name the gap)

Ask user to confirm or modify.

2. Select Evaluation Protocol

Present options with recommendation:

• □ K-fold cross-validation (recommend K=5 or K=10)
• □ Fixed train/val/test split
• □ Leave-one-out
• □ Episode-based evaluation
• □ Other (user specifies)

Justify the recommendation based on dataset size and domain convention.

3. Select Datasets

For each proposed dataset:

• Name and source (URL or citation)
• Why included (covers what aspect of the problem)
• Size and characteristics (samples, classes, difficulty)

Minimum: 2 datasets for Tier B venues, 3–5 for Tier A.

4. Define Seed Strategy

Recommend 5 seeds: [42, 123, 456, 789, 1024]

Present as default. User may modify but must keep ≥ 3 seeds. Fewer than 3 seeds is insufficient for statistical reporting and is not acceptable.

5. Define Statistical Reporting

Present options:

• □ mean ± std (minimum acceptable)
• □ mean ± std + 95% confidence interval (recommended)
• □ mean ± std + 95% CI + significance test (recommended for Tier A)