eval-testing

Eval Testing

Build comprehensive evaluation pipelines for AI agents using rule-based checks, LLM-as-judge scoring, golden datasets, and CI/CD integration.

When to Use

• Testing whether an agent produces correct, compliant output
• Evaluating RAG retrieval quality with RAGAS metrics
• Building a golden dataset for regression testing
• Setting up automated eval pipelines in CI/CD
• Comparing agent performance across prompt versions

Available Operations

1. Create rule-based evaluators for compliance checks
2. Build LLM-as-judge scoring prompts
3. Design golden datasets with expected outcomes
4. Run evaluation benchmarks
5. Integrate evals into CI/CD pipelines
6. Analyze and compare eval results

Multi-Step Workflow

Step 1: Create Rule-Based Evaluators

Start with deterministic checks -- they are fast, free, and reliable.

class ComplianceEvaluator:
    """Verify code follows bundle standards."""

    def evaluate(self, code: str, bundle: Bundle) -> EvalResult:
        checks = []
        checks.append(self._check_max_lines(code, max=500))
        checks.append(self._check_no_hardcoded_secrets(code))
        checks.append(self._check_function_length(code, max=20))
        checks.append(self._check_naming_convention(code))
        checks.append(self._check_test_coverage(code, min=80))

        score = sum(c.passed for c in checks) / len(checks)
        return EvalResult(score=score, checks=checks)

    def _check_max_lines(self, code: str, max: int) -> Check:
        lines = len(code.split('\n'))
        return Check(
            name="max_lines",
            passed=lines <= max,
            detail=f"{lines}/{max} lines"
        )

Run rule-based checks:

python -m evals.rules --code-dir src/ --bundle bundles/backend.json

Step 2: Build LLM-as-Judge Evaluator

For subjective quality assessments, use an LLM to score agent output.

JUDGE_PROMPT = """
Evaluate the code below on these criteria:
1. Clarity (1-5): Is the code easy to understand?
2. Correctness (1-5): Does the code do what it should?
3. Patterns (1-5): Does it follow Clean Architecture and DDD?
4. Tests (1-5): Are tests adequate?

Code:
{code}

Respond in JSON:
{{"clarity": X, "correctness": X, "patterns": X, "tests": X, "justification": "..."}}
"""

async def llm_judge(code: str) -> dict:
    response = await llm.ainvoke(JUDGE_PROMPT.format(code=code))
    return json.loads(response.content)