cekura-flag-call-log-failures

flag-call-log-failures

Go through an agent's recent production call logs and produce three things:

1. Flagged calls — the call IDs that hit each specified KPI/issue/goal failure, with verbatim evidence.
2. Failure rates — what percentage of all reviewed call logs each failure represents.
3. Outcome distribution — every remaining call bucketed into a mutually-exclusive outcome taxonomy (e.g. not-answered / vetted / non-vetted caller-side / non-vetted agent-issue), with per-bucket percentages, so the flagged failures are framed against the whole population.

It is the triage front-end: hand the flagged set to cekura-generate-scenarios to build evaluators, or give the user the distribution for a customer-facing quality report.

This skill is strictly read-only — it never creates, updates, or deletes anything.

The single most important job is attribution: a call is an agent failure only when the agent under test caused it. Caller-side endings, simulated-caller disconnects, and recovered calls must NOT be counted as agent failures — they belong in their own buckets.

---

Step 0 — Prerequisites

Reads through the Cekura MCP. Confirm these are present:

• mcp__cekura__aiagents_retrieve — agent description + intended behavior (the yardstick for "failure")
• mcp__cekura__call_logs_list — paginated production call list (lightweight: usually has id, duration, call_ended_reason, success, rolled-up metric scores — but often no transcript)
• mcp__cekura__call_logs_retrieve — full transcript + metric evaluations for a single call
• mcp__cekura__metrics_list — to find metrics that already grade the KPIs/issues (reuse these as the classification basis when they exist)

If the mcp__cekura__* tools aren't connected, stop and tell the user to connect the Cekura MCP (see /setup-mcp or https://docs.cekura.ai/mcp/overview).

---

Step 1 — Inputs

Use AskUserQuestion for anything not supplied:

1. Agent ID (numeric). If unknown, mcp__cekura__aiagents_list helps find it.
2. (Optional) Project ID.
3. Window — how many recent calls (default 100) or a date range. Process in batches (see Step 2).
4. KPIs / issues / goals to measure — the heart of the request:
   • Explicit issues — e.g. "calls that ended before all vetting questions", "agent looped on an unclear answer", "background noise stalled the agent". Each becomes a flag bucket + a failure-rate number.
   • A KPI / goal — e.g. "every call should fully vet the candidate", "what % of answered calls can we improve" — which you invert into the failure(s) that break it.
   • Nothing specific → "any genuine agent failure," grounded in agent_description, classified against the failure-mode taxonomy in Step 3.
5. Outcome taxonomy — the mutually-exclusive buckets to distribute ALL calls into. Derive these from the agent's job. For a candidate-vetting agent the natural set is:
   • not answered · vetted · non-vetted — caller-side · non-vetted — agent/system issue