agent-evaluation-reporting

Agent Evaluation Reporting

Overview

Turn raw agent evaluation runs into a decision-ready report without hiding failures or overstating capability. Keep outcome populations, denominators, latency populations, and experiment conditions explicit so readers can reproduce every headline number.

When to Use This Skill

• Use when reporting benchmark, regression, pilot, or production evaluation runs for an AI agent.
• Use when autonomous and human-assisted completions appear in the same result set.
• Use when failures, timeouts, infrastructure-invalid runs, retries, or partial results affect the denominator.
• Use when comparing two agents, prompts, harnesses, or releases and deciding whether the comparison is valid.

How It Works

Step 1: Freeze the comparison contract

Record the task set and sampling, model and provider, prompt or policy version, tool and harness versions, evaluator rubric, timeout and retry policy, token or cost budget, environment, and human-intervention policy. Assign the configuration a stable label or digest.

If a material condition differs between runs, mark the comparison as non-equivalent. Report a directional observation only; do not claim that the changed agent caused the difference.

Step 2: Build a mutually exclusive outcome ledger

Classify every scheduled attempt exactly once:

Outcome	Meaning
autonomous_success	The agent satisfied the evaluator without human intervention.
assisted_success	The task succeeded only after a human intervened.
failure	The run reached a terminal, evaluable failure.
timeout	The run exhausted its declared time or step budget.
invalid	The agent never received a valid evaluation because the harness, environment, or input failed.

Preserve attempt ID, task ID or seed, retry index, parent attempt ID, configuration label, outcome, intervention count, duration, cost, evaluator evidence, and invalid reason when available. Never silently drop invalid or retried runs.