eval-score

Eval — Score One Card

Run the LLM-as-judge against one daily briefing card and write the result to eval/store.sqlite.

How to invoke

Prefer the Makefile target. Default JUDGE is stub (offline heuristic, no API). Use JUDGE=claude for the real Claude Haiku 4.5 judge.

make eval D=YYYY-MM-DD                  # stub backend, no API cost
make eval D=YYYY-MM-DD JUDGE=claude     # real Claude judge (~$0.002/card)
make eval D=YYYY-MM-DD JUDGE=claude GATE=1  # also exit 2 if composite < 3.0

Equivalent direct invocation:

python3 eval/runner.py score --date YYYY-MM-DD --judge claude

Behavior

1. Read the card JSON at example-cards/YYYY-MM-DD-card.json (or logs/YYYY-MM-DD-card.json for fresh runs).
2. Pull the prior 7 days' headlines from example-cards/ as the novelty baseline.
3. Compose the judge prompt (eval/judge_prompt.md) + briefing text + prior headlines.
4. Send to the selected backend; the judge returns a JSON block with the 5 axis scores plus a notes field.
5. Compute composite = 0.30·F + 0.20·N + 0.15·D + 0.20·S + 0.15·C and upsert into eval_runs keyed on (card_date, prompt_version, judge_model).
6. Print the result as JSON. With --gate, exit 2 if composite is below --gate-threshold (default 3.0).

What to tell the user

Report the composite score, the per-axis breakdown, and the judge's notes verbatim — those notes usually call out the weakest axis with a concrete reason. If --gate is set and the run failed, surface that loudly along with the threshold.