qa-agent-testing

QA Agent Testing

Design and run reliable evaluation suites for LLM agents, including tool-using, multi-turn, and multi-agent systems.

Default QA Workflow

1. Define the Agent Under Test (AUT): scope, tools, approval boundaries, out-of-scope requests, and safety rules.
2. Build a starter suite from real work:
   • Smoke suite: 5-8 highest-signal checks for PR gates
   • Regression suite: 15-25 tasks from real failures, tickets, or production traces
   • Refusal/security pack: unsafe requests, prompt injection, tool-output poisoning, and exfiltration attempts
   • Iterative coding trajectory: evolving specifications applied to the agent's own carried workspace, when extension quality matters
3. Define objective graders first: schema checks, golden traces, deterministic mocks, policy oracles, and tool side-effect checks.
4. Add model-based graders only where objective checks are insufficient; calibrate them and log judge versions.
5. Run offline evals with deterministic controls and trace logging.
6. Add optional online evals or canary comparisons for live traffic.
7. Gate changes on one consistent status model and log regressions.

Use the starter templates in assets/ for day-0 setup. The template keeps 10 tasks + 5 refusals as a starter scaffold, not a best-practice cap.

Determinism and Flake Control

• Pin prompts, configs, fixtures, and tool mocks where possible.
• Freeze time, timezone, and locale for tests that depend on them.
• Log model, judge, and tool versions for every run.
• Record traces: prompt or message history, tool name, args, outputs, latency, errors, retries, approvals, and side effects.

Minimal instrumentation: Instrument agents at three points only — LLM call entry/exit (with span IDs), tool invocations (input, output, duration), and branching decision points (which path was chosen and why). Avoid instrumenting every intermediate computation; each additional trace dimension increases latency and storage cost, and the exact overhead depends on SDK, sampling, export path, and backend. Start minimal, expand only when a category of failure is consistently hard to diagnose without it.