agent-testing-patterns

Agent Testing Patterns

Test AI agent systems that use tools, make decisions, and produce non-deterministic outputs.

Testing Challenges

Challenge	Cause	Strategy
Non-deterministic output	LLM randomness	Assert on structure, not exact text
Tool use sequences	Agent autonomy	Verify tool calls, not call order
Multi-turn state	Conversation context	Snapshot-based assertions
Cost	API calls	Mock LLM in unit tests
Latency	API round-trips	Parallel test execution
Flakiness	Model updates	Semantic assertions, not string matches

Test Pyramid for Agents

        ╱╲
       ╱  ╲        E2E Agent Tests (few, expensive)
      ╱────╲       Full agent loop with real LLM
     ╱      ╲
    ╱────────╲     Integration Tests (moderate)
   ╱          ╲    Tool execution, state management
  ╱────────────╲
 ╱   Unit Tests ╲  Tool implementations, parsers, validators
╱────────────────╲

Unit Testing (No LLM)

Tool Implementation Tests

import pytest

def test_file_read_tool():
    tool = FileReadTool()
    result = tool.execute({"path": "test.txt"})
    assert result["content"] == "expected content"
    assert result["success"] is True

def test_file_read_tool_missing_file():
    tool = FileReadTool()
    result = tool.execute({"path": "nonexistent.txt"})
    assert result["success"] is False
    assert "not found" in result["error"].lower()

def test_tool_input_validation():
    tool = FileReadTool()
    with pytest.raises(ValueError, match="path is required"):
        tool.execute({})

Response Parser Tests

def test_parse_tool_call():
    raw = '{"tool": "search", "args": {"query": "python"}}'
    result = parse_tool_call(raw)
    assert result.tool == "search"
    assert result.args == {"query": "python"}

def test_parse_malformed_tool_call():
    raw = "not json at all"
    result = parse_tool_call(raw)
    assert result is None

Integration Testing (Mocked LLM)