skill-101

Agentic Evaluation Patterns

Patterns for self-improvement through iterative evaluation and refinement.

Workspace-baseline context: This skill supports the template quality workflow. Use alongside @Code Review Agent (quality checklist validation), @SpecKit Analyze (cross-artifact consistency), and the Quality Checklist in copilot-instructions.md to evaluate generated templates against baseline standards.

Overview

Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.

Generate → Evaluate → Critique → Refine → Output
    ↑                              │
    └──────────────────────────────┘

When to Use

• Quality-critical generation: Code, reports, analysis requiring high accuracy
• Tasks with clear evaluation criteria: Defined success metrics exist
• Content requiring specific standards: Style guides, compliance, formatting
• Template validation: Checking new instructions/agents/prompts/skills against baseline Hard Rules and Quality Checklist

---

Pattern 1: Basic Reflection

Agent evaluates and improves its own output through self-critique.

def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
    """Generate with reflection loop."""
    output = llm(f"Complete this task:\n{task}")
    
    for i in range(max_iterations):
        # Self-critique
        critique = llm(f"""
        Evaluate this output against criteria: {criteria}
        Output: {output}
        Rate each: PASS/FAIL with feedback as JSON.
        """)
        
        critique_data = json.loads(critique)
        all_pass = all(c["status"] == "PASS" for c in critique_data.values())
        if all_pass:
            return output
        
        # Refine based on critique
        failed = {k: v["feedback"] for k, v in critique_data.items() if v["status"] == "FAIL"}
        output = llm(f"Improve to address: {failed}\nOriginal: {output}")
    
    return output