bat-adhoc

BAT - Bot Acceptance Testing

Bot acceptance testing validates that MCP tools work correctly from a real AI agent's perspective. You design test scenarios dynamically, run them via tests/uat/run_uat.py, and evaluate results.

When to Use BAT

• PR validation: Test that tool changes work correctly from an agent's perspective
• Regression detection: Compare behavior between branches
• Integration verification: Ensure MCP tools work end-to-end with real agent CLIs

Workflow

1. Analyze the change: Read the diff, identify which tools are affected
2. Design scenario: Generate a scenario JSON with setup/test/teardown prompts
3. Run the script: Pipe the scenario to python tests/uat/run_uat.py
4. Evaluate summary: Check all_passed per agent. If true, you're done.
5. Dig deeper on failure: Read results_file for full output, stderr, raw JSON
6. Regression check: If test fails, re-run with --branch master to compare

Output Structure

The runner returns a concise summary to stdout (saves context when all passes):

{
  "results_file": "/tmp/bat_results_abc123.json",
  "agents": {
    "gemini": {
      "all_passed": true,
      "test": {
        "completed": true,
        "duration_ms": 8100,
        "exit_code": 0,
        "num_turns": 5,
        "tool_stats": { "totalCalls": 4, "totalSuccess": 4, "totalFail": 0 }
      },
      "aggregate": {
        "total_duration_ms": 15300,
        "total_turns": 12,
        "total_tool_calls": 9,
        "total_tool_success": 9,
        "total_tool_fail": 0
      }
    }
  }
}

• Phase stats: num_turns, tool_stats (per phase) for fine-grained comparison
• Aggregate stats: Total counts across all phases for overall efficiency comparison
• On failure: also includes output and stderr for diagnosis
• Full results: raw JSON, complete output always available at results_file

Scenario Design Guidelines

• setup_prompt: Create any entities/state the test needs
• test_prompt: Exercise the tools being tested, ask the agent to report results clearly
• teardown_prompt: Clean up created entities
• Keep prompts focused - each scenario tests ONE behavior
• Ask the agent to report: what succeeded, what failed, any unexpected behavior