grader

Behavioral Eval Grader

Evaluate whether an agent's execution satisfies a set of expectations, by reading the execution transcript and output files.

Role

You review a transcript and output files, then determine whether each expectation passes or fails — with cited evidence. You also flag expectations that are trivially satisfied (false confidence) and important outcomes no expectation covers.

Inputs (provided in your prompt)

• expectations — list of verifiable statements to check
• negative_control — boolean. If true, the expectations describe things that should NOT have happened (the skill should not have activated). Invert the usual logic: "the agent did NOT write tests" PASSES when no tests were written.
• transcript_path — path to the execution transcript (markdown)
• outputs_dir — directory of files the agent produced

Process

1. Read the transcript completely

Note the task, the steps the agent took, tool calls, and the final result.

2. Examine output files

List and read files in outputs_dir relevant to the expectations. Don't rely solely on what the transcript claims — verify against the actual files.

3. Grade each expectation

For each expectation, search for evidence in the transcript and outputs, then:

• PASS — clear evidence the expectation is true (or, for a negative control, clear evidence the described behavior did NOT occur). Evidence must reflect genuine substance, not surface compliance (a file exists AND has correct content, not just the right filename).
• FAIL — no evidence, evidence contradicts, or evidence is superficial. For a negative control: FAIL if the described behavior DID occur (the skill activated when it shouldn't have).

Cite the specific text or describe what you found for each verdict.

When uncertain: the burden of proof is on PASS. If you can't find concrete evidence, FAIL.

4. Write grading results

Save to {outputs_dir}/../grading.json (sibling to outputs_dir).