eval-answer

Evaluate One Answer

One user intent, answered once. This skill decides whether that answer was correct, records the evidence, and stops.

Scope boundary: verdict and events only. No diagnosis, no model edit.

The unit

A chat is not the unit. Segment by user intent. Feedback ("break it out by region") is a revision inside the same answer; grade the final accepted revision.

Take the question from the stored case (evals/<set>/cases.jsonl), never from memory or a truncated console line. Record question_sha of the exact text the answerer saw. Record servedRevision from get_context or reload, not the package name: a same-named decoy has been measured for hours.

Step 1: Contamination check, before any score

The answerer can Read or Shell its way to gold. Publisher traces do not see that, so the check runs on the HOST-side tool-use log you kept for the answerer subagent (every tool name and its path or command), plus the MCP call counts the answerer reported.

The checklist. An attempt is contaminated when its log shows any of:

1. a Read, Shell, or any file tool touching evals/ or a gold artifact path;
2. any access to the model file under test through a file tool (the modelPath argument on an MCP execute_query is NOT contamination; the server resolves it, the answerer never reads the file);
3. reported_calls greater than host_tool_uses (the detectable under-report floor is reported at most total tool uses).

skills/eval-answer/scripts/check_contamination.py is a reference aid that mechanizes the same checklist over a JSON log; your reading of the transcript is the check, the script is a second pair of eyes.

Contaminated attempts get verdict: null and contaminated: true. They are excluded from the run aggregates. They are not "wrong answers."

If you cannot produce a host log, mark contaminated: "unknown" on both the attempt and its score event, and do not treat the attempt as a clean pass.

Step 2: Re-run the submitted query yourself