eval-dataset-design

Eval Dataset Design

Your evals are only as good as the dataset they run on. Miss a user scenario and you'll never catch regressions on it.

When to Use

• Starting an eval program from zero
• Your evals pass but users still hit issues → coverage gap
• Labels are inconsistent across reviewers → quality problem
• Adding evals for a new feature or domain

Dataset Properties Worth Optimizing

1. Coverage — representative of real user queries
2. Difficulty distribution — mix of easy/medium/hard, not all easy
3. Label consistency — two humans agree on the label
4. Stability — same inputs → same evaluable outputs over time
5. Uncontaminated — not in the model's training data

Sourcing Inputs

Best to worst:

1. Real user queries (anonymized) — highest signal
2. Synthetic queries generated from real templates — fills gaps
3. Adversarial queries hand-crafted for known failure modes
4. Existing benchmarks — context, but often contaminated and dated

A good eval set mixes all four. Typical split: 60% real, 20% synthetic, 15% adversarial, 5% benchmark.

Stratification

Split your dataset by categories that matter:

dataset:
  categories:
    simple_qa: 100 samples        # easy, high-frequency
    multi_step_reasoning: 50       # medium
    ambiguous_queries: 30          # hard
    edge_cases: 20                 # adversarial
    rare_domains: 20               # coverage of long tail

Report metrics per stratum, not just the aggregate. A model can improve on average while regressing on edge cases — you'll only see it stratified.

Labeling Quality

Two people label the same 50 items independently. Compute inter-annotator agreement:

from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(labeler_a, labeler_b)

Target:

• κ > 0.8: excellent, labels are reliable
• κ 0.6-0.8: good, some ambiguity
• κ < 0.6: rewrite your labeling rubric — humans can't agree, so neither can models

Resolve disagreements with a tiebreaker, then update the rubric based on what caused disagreement.