eval-answer

Evaluate One Answer

One user intent, answered once. This skill decides whether that answer was correct, records the evidence, and stops.

Scope boundary: verdict and events only. No diagnosis, no model edit.

The unit

A chat is not the unit. Segment by user intent. Feedback ("break it out by region") is a revision inside the same answer; grade the final accepted revision.

Take the question from the stored case (evals/<set>/cases.jsonl), never from memory or a truncated console line. Record question_sha of the exact text the answerer saw. Record servedRevision from get_context or reload, not the package name: a same-named decoy has been measured for hours.

Step 1: Contamination check, before any score

The answerer can Read or Shell its way to gold. Publisher traces do not see that, so the check runs on the HOST-side tool-use log you kept for the answerer subagent (every tool name and its path or command), plus the MCP call counts the answerer reported.

The checklist. An attempt is contaminated when its log shows any of:

1. a Read, Shell, or any file tool touching evals/ or a gold artifact path;
2. any access to the model file under test through a file tool (the modelPath argument on an MCP execute_query is NOT contamination; the server resolves it, the answerer never reads the file);
3. reported_calls greater than host_tool_uses (the detectable under-report floor is reported at most total tool uses). host_tool_uses is EVERY tool use the host logged, MCP calls included; while it counted only the non-MCP ones this comparison was true of almost every clean attempt, so a run whose attempts predate the split cannot be checked this way.

skills/eval-answer/scripts/check_contamination.py is a reference aid that mechanizes the same checklist over a JSON log; your reading of the transcript is the check, the script is a second pair of eyes.

Contaminated attempts get verdict: null and contaminated: true. They are excluded from the run aggregates. They are not "wrong answers."