agent-evaluation

Agent Evaluation

Activate this skill when building or fixing the harness that decides whether an agent change is safe to ship.

When to Use

• Standing up an eval harness for a new agent
• Wiring an eval job as a CI merge gate
• Configuring a Bedrock evaluation job
• Diagnosing a quality regression, or building online drift alarms

Structure

evals/
  golden/
    support.jsonl          # {id, input, expected_tools, expected_facts, rubric}
    adversarial.jsonl      # injection, out-of-scope, unanswerable, PII-bait
  runners/
    run_offline.py         # execute the golden set, emit results.json
    judge.py               # LLM-as-a-Judge with a pinned model + rubric
  thresholds.yaml          # gate config — the only place numbers live
  reports/

Golden Sets

• 50–200 cases minimum before a pass rate means anything; below that a two-case swing is noise.
• Mine cases from production transcripts, not imagination. Every incident adds a case — that is what stops the same regression twice.
• Stratify by intent, difficulty, and known failure mode, and report per stratum. Aggregate scores hide the regression that matters.
• Always include: prompt-injection attempts, out-of-scope questions, questions the corpus cannot answer (correct response is a refusal), and PII-bait.
• Version the set with the code; never change the set and the system in one commit.

Deterministic Checks First

Anything checkable in code is checked in code — faster, cheaper, and not itself a model that can drift:

assert result.tools_called == case["expected_tools"]        # routing
assert Schema.model_validate(result.output)                 # structure
assert all(c.source_id in result.retrieved_ids
           for c in result.citations)                       # citations are real
assert result.latency_p95 < budget
assert result.tokens_total < budget
assert result.guardrail_triggered == case["expect_guardrail"]

Tool-selection accuracy is the highest-signal cheap metric in an agent system. Most "the agent gave a bad answer" reports are "the agent called the wrong tool".