agent-kernel: Skill for Claude Code

Adding a New Test Evaluator Provider

This guide walks through adding a new built-in evaluator provider to Agent Kernel's test framework. Use the existing DeepEval implementation (ak-py/src/agentkernel/test/core/evaluator/deepeval.py) as reference.

Before starting, check whether you actually need this skill: if the evaluator only needs to exist for your own project (not addressable by every AK user via a short built-in name), you don't need any of the steps below — just subclass AKEvaluator anywhere importable and point test-config.yaml's evaluator: at its dotted path. That's the "bring your own evaluator" path described in the user-facing ak-test skill and docs/docs/testing/cli-testing.md; examples/cli/custom-evaluator/ is a complete worked example of it. This skill is only for adding a first-party, in-repo provider that ships with AK and gets its own short type name.

Existing Providers

Architecture Overview

• AKEvaluator (ak-py/src/agentkernel/test/core/evaluator/base.py) is the abstract base every evaluator — built-in or bring-your-own — implements. It and the payload and error types are re-exported from agentkernel.test, the import path users and examples use; built-in providers inside the package import them from .base. A new interface type goes in core/evaluator/__init__.py's __all__ and in agentkernel/test/__init__.py. It has exactly two abstract methods:
  • evaluate_by_score(case: AKEvaluationCase) -> AKEvaluationResult — deterministic scoring, no LLM call
  • evaluate_by_llm(case: AKEvaluationCase) -> AKEvaluationResult — LLM-as-judge scoring
• AKEvaluationCase carries the comparison inputs (user_input, actual, expected, threshold, context, criteria); AKEvaluationResult carries the outcome (score, passed, metric, evaluator, reason, cost, attempts, metadata).
• Error contract (every evaluator must honor this, not just DeepEval):
  • Raise AKMissingInput when a field the requested metric needs (e.g. case.expected) wasn't supplied.
  • Raise AKMetricNotSupported from whichever of the two methods your backend structurally cannot implement (e.g. a pure LLM-judge service has no offline scoring mode).
  • Raise AKEvaluationError when a configured backend fails to produce a score (missing credentials, transport error, unparseable judge output). Never raise AssertionError and never silently return a 0.0 to stand in for a failure — 0.0 must only ever mean "scored zero", not "couldn't be scored". Test.compare is the only place that decides pass/fail fatality; evaluators only ever set result.passed.
• Test._resolve_evaluator_class in ak-py/src/agentkernel/test/test.py is the factory. It shares the same pluggable-backend shape as guardrails, sandbox providers, and trace backends (core/util/factory.py's resolve_dotted/require_extra/AKConfigError): an if-per-built-in branch with the SDK import wrapped in require_extra (actionable ImportError naming the pip extra if missing), then a dotted-path bring-your-own fallback for anything else.
• Evaluator instances are cached per-process on Test._evaluator (keyed by the configured value), guarded by Test._evaluator_lock — construction happens once per distinct evaluator: config value, not once per Test.compare call.