agent-eval

When to invoke

Use when: "evaluate the agent", "build an eval set", "LLM as judge", "prompt regression test", "did the prompt change make it better", "measure prompt quality", "eval harness".

Do not use for web page performance (/benchmark) or for choosing between vendors on a single prompt (/benchmark-models). This skill builds a repeatable test suite for one unit of LLM behavior in the user's own project.

Preamble

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
  _LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
  echo "LEARNINGS: $_LEARN_COUNT entries loaded"
  if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
    ~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
  fi
else
  echo "LEARNINGS: none yet"
fi

{{include lib/snippets/session-host.md}}

{{include lib/snippets/decision-brief.md}}

{{include lib/snippets/working-protocols.md}}

{{include lib/snippets/state-protocols.md}}

User-invocable

When the user types /agent-eval, run this skill. An optional argument names the unit under test (a file path, a function, or a prompt name); without one, find it in Step 1.

Step 1: Identify the unit under test

Pin down exactly one thing to evaluate. An eval that covers "the whole agent" measures nothing.

Find candidates with Grep and Glob:

• Prompt templates: **/*prompt*, **/prompts/**, string literals containing You are or system:
• Agent loops: files that call a chat completion API in a loop and dispatch on tool calls
• Tool routers: a function that maps a model's tool-call name to a handler
• Existing evals or fixtures: **/eval*, **/golden*, **/fixtures/**, *.jsonl, transcripts under logs/

For the chosen unit, write down its interface before writing any case: