audit-connector

Audit Connector Workflow

You are auditing a built connector the way Anthropic's Petri audits a model: real agent tasks, parallel auditors, transcripts as the signal, and a judge that scores graded dimensions with cited evidence. The connector must already be built (elliot_build_connector) — the audit runs against it.

Steps

1. Generate audit seeds

Call elliot_generate_audit_seeds (count defaults to 5). It returns seeds (realistic agent tasks derived from the product intent and the tools), the scoring rubric, and the spawn instructions.

2. Spawn 5 parallel sub-agents

Spawn one sub-agent per seed in parallel (a single message with multiple Task calls). Brief each sub-agent like this:

You are an AI agent whose ONLY tools are this connector's tools. Accomplish this task: <seed.task>. Exercise each tool by calling elliot_preview_tool with the tool id and arguments — this runs the tool against the sandbox data. If the connector ships skills (multi-step workflows) and your task matches one, use it: call elliot_preview_skill(skill_id, inputs={...}) — it runs the whole chain in one call, exactly as an agent would invoke the published skill-tool, and it's the point of the skill. Prefer a matching skill over hand-chaining its steps. Pick argument values from the parameter descriptions; do not guess blindly. Follow the rubric. Return a transcript: for every call (tool OR skill) record tool_id (the skill id for a skill call, with is_skill: true), arguments, ok, error_code, error_message, result_row_count, result_token_estimate, and a short note when something was confusing. Also return task_completed (bool) and a one-line summary.

The sub-agents run against the in-memory sandbox — the audit is safe and makes no real API calls.

3. Submit transcripts

For each sub-agent, call elliot_submit_audit_transcript with its transcript JSON (seed_id, task, agent_label, calls, task_completed, summary).