ak-test

Testing & Debugging

Use this skill to set up testing for your Agent Kernel project or debug issues.

Instructions for the Agent

Setting Up Tests

1. Add Test Dependencies

Update pyproject.toml:

[dependency-groups]
dev = [
    "agentkernel[test]>=0.8.1",
    "black>=23.0.0",
    "isort>=5.0.0",
    "mypy>=1.0.0",
]

Run uv sync to install test dependencies.

2. Choose a Test Mode

Update config.yaml:

test:
  mode: fuzzy       # Options: fuzzy | judge | fallback

Mode	How it Works	Best For
fuzzy	String similarity matching (rapidfuzz)	Deterministic responses, exact answers
judge	LLM evaluates if response is semantically correct	Open-ended responses, creative agents
fallback	Tries fuzzy first, falls back to judge if fuzzy fails	General-purpose testing

For judge mode, configure the judge model:

test:
  mode: judge
  judge:
    model: gpt-4o-mini

3. Write CLI Agent Tests

For agents running via CLI (demo.py):

import pytest
import pytest_asyncio
from agentkernel.test import Test

pytestmark = pytest.mark.asyncio(loop_scope="session")


@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def test_client():
    test = Test("demo.py")       # Path to your agent definition file
    await test.start()
    try:
        yield test
    finally:
        await test.stop()


@pytest.mark.order(1)
async def test_greeting(test_client):
    await test_client.send("Hello!")
    await test_client.expect(["Hello", "Hi", "Greetings"])


@pytest.mark.order(2)
async def test_specific_question(test_client):
    await test_client.send("What is the capital of France?")
    await test_client.expect(["Paris"])


@pytest.mark.order(3)
async def test_follow_up(test_client):
    # Follow-up questions work because session state is maintained
    await test_client.send("What is its population?")
    await test_client.expect(["2 million", "2.1 million", "approximately 2 million"])