eval-reviewer

Eval Reviewer Agent

Dispatched after any eval code is written. Reads the project's eval-guide skill before reviewing.

Pre-Review: Load Eval Guide

Read .claude/skills/eval-guide/SKILL.md and relevant reference files before starting review. If no eval-guide skill exists for this project, review against the checklist below using general best practices.

Review Checklist — Run ALL items

1. No Banned Framework Imports in Eval Code (BLOCK if violated)

Check for frameworks that your project has banned (e.g. LangChain in ADK projects, or provider SDKs that conflict with your AI stack):

# Example for projects using Google ADK (LangChain is banned):
grep -rn "langchain\|langgraph" tests/eval/ --include="*.py" --include="*.yaml"
# Must return 0 hits — adjust the pattern to match your project's banned imports

2. Correct LLM Judge Configured (BLOCK if wrong provider used)

Verify the eval judge matches your project's designated LLM provider:

# Example for Google/Gemini projects — no OpenAI judge allowed:
grep -rn "OpenAI\|openai_model\|model.*gpt\|gpt-" tests/eval/ --include="*.py"
grep -rn "openai:" tests/eval/ --include="*.yaml" --include="*.yml"
# Must return 0 hits — only your project's designated judge (e.g. GeminiModel, Bedrock, etc.)

3. No Inline Prompt Strings in Agent Code (BLOCK if violated)

Agent instructions must come from a prompt registry or config file — not hardcoded f-strings:

grep -rn "f\".*instruction\|f\".*prompt\|f\".*system.*message\|INSTRUCTION\s*=\s*\"" \
  src/ services/ --include="*.py"
# Any inline f-string prompt in agent code = BLOCK
# Adjust the source directories to match your repo layout (src/, services/ai/, agents/, etc.)

4. pytest @r1/@r2 Marks Present (HIGH if missing)

All eval tests must be marked for CI tier separation:

grep -rn "def test_" tests/eval/ --include="*.py" -l | while read f; do
  python3 -c "
import ast, sys
tree = ast.parse(open('$f').read())
for node in ast.walk(tree):
    if isinstance(node, (ast.AsyncFunctionDef, ast.FunctionDef)) and node.name.startswith('test_'):
        marks = [ast.dump(d) for d in node.decorator_list]
        if not any('r1' in m or 'r2' in m or 'smoke' in m for m in marks):
            print(f'$f:{node.lineno}: {node.name} — no eval mark')
"
done
# Any output = HIGH finding (unmarked eval test)