gcf CLAUDE.md

GCF Project Instructions

Eval Data Update Procedure

When new comprehension or generation eval runs are added, the following files must be updated. Do all of them in a single pass. Do not wait to be asked.

CRITICAL: Never copy numbers from existing docs or memory. Always recompute from the raw log files. Stale numbers have propagated before because averages were copied from one surface to another instead of recomputed. The log files in eval/results/comprehension/ are the single source of truth.

Step 0: Verify from raw logs

Before updating anything, recompute all headline numbers from the raw log files:

# Run this from eval/results/comprehension/
# Parse ALL comprehension-13q-*.log AND any .txt files with comprehension data
# Extract per-format scores, group by model, compute per-model averages
# Then average the per-model averages (NOT the per-run averages)
# This is the "average of model averages" methodology

Known naming inconsistencies in log files:

• Some runs use hyphens (gemini-2.5-flash), others don't (gemini25flash). Same model.
• Some runs are .txt not .log (e.g., haiku-4.5-run2.txt). Still valid data.
• Opus has run2 but no run1 comprehension log. Only 1 adversarial run exists.
• Always glob for both comprehension-13q-* AND any other .txt files with comprehension data.

Step 1: Record the run

• Save log file to eval/results/comprehension/ or eval/results/generation/ with standard naming: comprehension-13q-{model}-run{N}-{date}.log or generation-{model}-run{N}-{date}.log

Step 2: Update SUMMARY (this repo)

• eval/results/SUMMARY.md: Add to "All runs" table, update "Averages by model" table, update file listing. If failure taxonomy counts change, update those too.

Step 3: Recalculate headline numbers

After updating the SUMMARY, recalculate FROM RAW LOGS (not from existing docs):

• Overall GCF average: average of per-model averages (not average of all runs; models with more runs should not be overweighted)
• Run count: total comprehension runs, total generation runs
• Win/tie/loss record: GCF vs TOON across all comprehension runs
• Total evaluations: (comprehension runs x 13 questions x 3 formats) + (generation runs x 5 sizes x 3 formats)
• Model count: count of distinct models (combine same-model runs with different naming)