regression-test

Regression Test

Purpose

Catch behaviour regressions with simulated callers before real customers hit them. A simulation is an AI caller with a persona prompt who talks to the agent for up to N turns; each case passes or fails against its success criteria.

The loop:

plan cases → build or reuse a suite → run → collect results → report

Use it before publishing a non-trivial change (see the safe-agent-change skill), after publishing to confirm a fix, or to reproduce a failure found by the call-review skill.

This playbook runs on the Synthflow MCP tools, whether they come from the plugin's bundled server or the official Synthflow connector. Simulations place no phone calls.

Tools this uses

• list_agents, get_agent, get_agent_actions — the agent under test, its prompt, and what it can do.
• list_simulation_suites, list_simulation_cases — reuse existing tests before writing new ones.
• create_simulation_suite — name, model_id, language (default en-US).
• create_simulation_case — name, prompt (the caller persona), success_criteria (list), call_success_type (all or any), suite_id.
• execute_simulation_suite — suite_id, target_agent_id, max_turns (10–50, default 20). Returns a session id.
• get_simulation_session — status, transcripts, and per-criterion results for that run.
• list_simulations — past runs, filterable by agent, session, and status.
• get_call_details — when turning a real failed call into a test case.

Step 0 — Pin the target

1. Agent and workspace. Resolve the exact model_id (and subaccount workspace_id if needed, via list_subaccounts). Never guess IDs.
2. What version is being tested. Simulations run against the agent's configuration as Synthflow serves it. Don't assume a run picks up an unpublished draft. To test a change before it reaches production, apply it to a test copy of the agent (a separate agent with the same prompt and actions) and run the suite against that copy.
3. Side effects. Simulated conversations can trigger the agent's attached actions — bookings, CRM writes, SMS, webhooks. Check get_agent_actions. If the agent has actions that write to real systems, tell the user before running, and recommend a test copy with those actions detached or pointed at a sandbox.