agentic-context-engine: Skill for Claude Code

Stage 3: Metrics and Programmatic Analysis

Define metrics from insights, implement as code, run, review, iterate.

Inputs

• TRACES_FOLDER — path to directory containing trace JSON files
• eval/stage1_insights_summary.md — output from Stage 1
• eval/stage2_domain_context.md — output from Stage 2

Read both input files before starting.

Process

This stage is iterative. You cycle through define → implement → run → review, with a hard cap of 3 iterations. A metric set is "clean" when ALL of the following hold:

1. No small-sample metrics in the priority set — every metric used for priority ranking has denominator >= 5. Metrics with denominator < 5 are kept but labeled "confidence": "directional-only" and excluded from priority sorting.
2. No unexplained extremes — no metric reads 0% or 100% unless you can write a one-sentence justification (e.g., "0% is correct because the agent never calls send_certificate anywhere in the dataset"). Record the justification in the metric's "extreme_justification" field.
3. No redundant pairs — no two metrics share > 70% of their denominator events. Check this: for each pair, compute |events_A ∩ events_B| / min(|events_A|, |events_B|). If > 0.70, merge or drop one.
4. Script runs without errors on the full trace set.

If after 3 iterations the set is not fully clean, ship what you have and log remaining issues in eval/baseline_metrics.json under a top-level "warnings" key.

Step 1: Define metrics

For each insight from the Kayba analysis, use the evidence fields to identify observable signals in the traces:

1. Read the insights summary — focus on evidence citations, error strings, behavioral patterns
2. For each valid insight, determine what trace signal would change if the agent followed the skill
3. Classify each metric by detector pattern type:

Recovery detectors — consecutive calls to the same function where first has error, next succeeds

def has_recovery(calls, function_name):
    for i in range(len(calls) - 1):
        if calls[i]['name'] == function_name and is_error(calls[i]['output']):
            if calls[i+1]['name'] == function_name and is_success(calls[i+1]['output']):
                return True
    return False