call-goal-drift-auditor

call-goal-drift-auditor

An agent told to confirm an appointment that ends up selling whitening packages has drifted. Nobody noticed. Until now.

Every CALL-E call starts with a goal. But calls are dynamic: a curious callee, an ambitious agent script, or a distraction tangent can pull the conversation far from the stated objective. Without post-call auditing, nobody knows the goal was never really achieved — or that three agent turns were spent on topics that were explicitly out of scope.

This skill reads the finished transcript alongside the original goal text, extracts content keywords, measures per-turn on-topic ratio, detects off-topic spans, and reports whether the goal appears achieved in the closing confirmation. It also generates a tighter, bounded goal with explicit scope-limiting instructions for the next call.

When To Use

• after any CALL-E call where scope discipline matters
• when a call goal is narrow and violations are measurable (appointment confirmation, payment verification, consent capture)
• as part of a QA pipeline to detect systematic agent scope creep
• before replaying a failed or incomplete call with tighter instructions

When Not To Use

• without a goal file; drift is measured against the goal text and the CLI requires it
• when the goal is intentionally broad ("assist the caller as needed")
• as a legal or compliance ruling; it is heuristic and advisory only
• for non-English transcripts; keyword extraction is English-only

Workflow

Audit a finished call

python3 scripts/goal_drift_auditor.py analyze \
  --transcript path/to/call-result.json \
  --goal-file path/to/goal.txt

Reads the real get_call_run result shape or flat fixture shape. The goal file may be plain text or a JSON object with a goal, task, or objective key. Emits a drift card:

• goal_keywords[]: content words extracted from the goal (sorted, 4+ char, non-stop-word)
• agent_turn_count / on_topic_turn_count / on_topic_ratio
• off_topic_spans[]: each span of ≥ 2 consecutive agent turns without goal keywords, with start_turn_index, length_in_agent_turns, and PII-masked first_off_topic_evidence
• goal_achieved: true if goal keywords appear with a confirmation phrase in any of the last 4 agent turns
• verdict: ON_TRACK / MILD_DRIFT / SIGNIFICANT_DRIFT / GOAL_NOT_ACHIEVED, plus unclear paths
• recommended_action: no_action_required, monitor_and_consider_tighter_goal, or retry_with_tighter_goal with guidance