metric-design

Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose the right tool for each evaluation dimension — from zero-cost code checks to LLM judges — and produce executable GradingRunner code that runs on OpenJudge.

Requires OpenJudge (pip install py-openjudge). This skill is intentionally SDK-centric — grader selection, GradingRunner, and aggregators are OpenJudge APIs. The design/decision logic still applies if you use another harness; only the code does not.

When to Activate

• User has eval dimensions/principles but doesn't know which grader type to use
• User wants to write an LLM-as-judge prompt for a specific failure mode
• User needs a composite score combining multiple evaluation dimensions
• User wants to auto-generate graders from labeled data instead of writing them manually
• User's current evaluation is all LLM-based and too expensive/too slow

Checklist

You MUST create a task for each item and complete them in order:

1. Select grader types — per dimension, pick the right grader class
2. Create custom graders — write judge prompts (4-component) or function graders
3. Auto-generate if applicable — use OpenJudge Generator for cold starts
4. Run anti-pattern scan — check for Likert, missing few-shot, vague criteria
5. Build pipeline code — assemble GradingRunner with graders + aggregators

Step 1: Select Grader Type Per Dimension

For each evaluation dimension, walk this decision tree (first match wins):

1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence

2. Does it require semantic understanding of text quality?
   → LLMGrader with built-in class (low cost, pre-optimized)
   Examples: CorrectnessGrader (factual match), RelevanceGrader (on-topic check),
             HallucinationGrader (faithfulness to context)

3. Does it involve agent behavior (tool calls, planning, memory)?
   → Agent-specific LLMGrader
   Examples: ToolSelectionGrader, TrajectoryAccuracyGrader, MemoryAccuracyGrader

4. Does it involve code execution or syntax?
   → CodeExecutionGrader / SyntaxCheckGrader
   Examples: test case pass rate, syntax validity, code style checks

5. Does it require external tool calls to verify (web search, database lookup)?
   → AgenticGrader (expensive, use only when necessary)
   Examples: fact-checking against live sources, cross-referencing databases