agent-evaluation

Agent Evaluation Skill

Evaluate AI agent task execution using world-class LLM-as-judge patterns from DeepEval, RAGAS, and G-Eval frameworks.

Output Format

Evaluation results are saved to evals/results/eval-${yyyy-mm-dd-hh-mm}-${commit_id}.md

Results Table

Task Input	Agent Output	Reflection Input	Reflection Output	Score	Verdict	Feedback
Create hello.js...	I've created hello.js with...	Task: Create hello.js Agent Output: ...	Task complete	5/5	COMPLETE	Agent produced output; Found completion indicators
Fix the bug...	I found the issue and...	Task: Fix bug Agent Output: ...	(none)	3/5	PARTIAL	Agent produced output; Missing reflection

Run Evaluation

# Run E2E evaluation
npx tsx eval.ts

# Or via npm
npm run eval:e2e

# Output saved to: evals/results/eval-2026-01-28-12-30-abc1234.md

---

Evaluation Rubric (0-5)

Score	Verdict	Criteria
5	COMPLETE	Task fully accomplished. All requirements met. Optimal execution.
4	MOSTLY_COMPLETE	Task done with minor issues. 1-2 suboptimal steps.
3	PARTIAL	Core objective achieved but significant gaps or errors.
2	ATTEMPTED	Progress made but failed to complete. Correct intent, wrong execution.
1	FAILED	Wrong approach or incorrect result.
0	NO_ATTEMPT	No meaningful progress. Crashed or no output.

Pass threshold: >= 3 (development), >= 4 (production)

---

Evaluation Prompt Template

Use this prompt for LLM-as-judge evaluation:

You are an expert evaluator assessing AI agent task completion.

## Original Task
{{task}}

## Execution Trace
{{trace}}

## Final Output
{{output}}

## Evaluation Criteria
1. Was the core objective achieved?
2. Were appropriate tools selected?
3. Were tool arguments correct?
4. Was execution efficient (minimal steps)?
5. Is the final output accurate and complete?

## Scoring Rubric
- 5: COMPLETE - All requirements met perfectly
- 4: MOSTLY_COMPLETE - Minor issues only
- 3: PARTIAL - Core done but significant gaps
- 2: ATTEMPTED - Progress made but failed
- 1: FAILED - Wrong approach or result
- 0: NO_ATTEMPT - No meaningful progress

## Instructions
1. Analyze the execution step-by-step
2. Identify specific issues or strengths
3. Score using the rubric
4. Provide actionable recommendations

## Response Format (JSON only)
{
  "reasoning": "<step-by-step analysis>",
  "score": <0-5>,
  "verdict": "<COMPLETE|MOSTLY_COMPLETE|PARTIAL|ATTEMPTED|FAILED|NO_ATTEMPT>",
  "feedback": "<1-2 sentence summary>",
  "recommendations": ["<improvement 1>", "<improvement 2>"]
}