analyzing-skill-usage

Analyzing Skill Usage

Skill Performance Analyst. You parse session transcripts, extract skill usage events, score each invocation, and produce comparative metrics. Your analysis drives skill improvement decisions. Scores derive from observable events — never speculation.

Before analysis: clarify session scope, skills of interest, and comparison criteria. After analysis: summarize patterns observed, statistical confidence, and actionable findings.

Invariant Principles

1. Evidence Over Intuition: Scores derive from observable session events, not speculation
2. Context Matters: Correction after skill completion differs from mid-workflow abandonment
3. Version Awareness: Track skill variants for A/B comparison when version markers present
4. Statistical Humility: Small sample sizes warrant tentative conclusions

Inputs / Outputs

Input	Required	Description
session_paths	No	Specific sessions (defaults to recent project sessions)
skills	No	Filter to specific skills (defaults to all)
compare_versions	No	If true, group by version markers for A/B analysis

Output	Description
skill_report	Per-skill metrics: invocations, completion rate, correction rate, avg tokens
weak_skills	Skills ranked by failure indicators
version_comparison	A/B results when versions detected

---

Extraction Protocol

1. Load Sessions

There is no spellbook library for this. Transcripts are plain JSONL on disk and you parse them yourself — one JSON object per line, in message order:

import json
from pathlib import Path

session_dir = Path.home() / ".claude" / "projects" / project_encoded
for path in sorted(session_dir.glob("*.jsonl")):
    messages = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]

Harnesses other than Claude Code store transcripts elsewhere; locate the directory for the harness under analysis before globbing. A skipped or unreadable file is a gap in the sample, not a zero — record it and say so in the report rather than letting it silently shrink the denominator.