agent-eval-workflow

Agent evaluation: the process

Commands change. The process does not. This skill is about how to evaluate an agent well — the reasoning that stays true whether you drive it with agent-eval, agents-cli eval, or whatever replaces them.

The loop: Hypothesize. Test. Validate.

State a falsifiable claim about the agent, write metrics that could prove you wrong, run them, and let the results — not your intuition — decide what to fix.

---

1. Start from a hypothesis, not from "add some metrics"

A metric exists to settle an argument. Before generating anything, answer:

• What do I believe this agent gets wrong?
• What observable evidence would prove that?
• What would prove me wrong?

Read the agent's own instructions and tool code first. Stated rules are testable claims — "always confirm before deleting", "never approve above 10%", "always cite a source". Those sentences convert directly into criteria.

When a generator asks what to focus on, give it the hypothesis in plain language naming real tools and thresholds. Vague guidance produces vague rubrics.

Good: "The agent must never call approve_discount above 10%; it should route larger requests to sync_ask_for_approval instead of being rejected and retrying."

Weak: "Test discount handling."

Prefer binary rubrics (0/1). LLM judges have poor inter-rater reliability on 1–5 scales. Several sharp binary metrics beat one fuzzy graded one, and the pass rate across a dataset gives you a continuous score with better statistics.

---

2. Make the agent measurable before measuring harder

A judge reading prose is fuzzy. A state variable written by the tool itself is deterministic. If a tool already makes a decision, record it:

def approve_discount(discount_type: str, value: float, reason: str,
                     tool_context: ToolContext) -> dict:
    if value > MAX_DISCOUNT_RATE:
        tool_context.state["discount_status"] = "rejected"   # hard evidence
        return {"status": "rejected", "message": "discount too large."}
    tool_context.state["discount_status"] = "approved"
    return {"status": "ok"}