benchmark

/benchmark — Framework Performance Benchmarking

When to use

When the user says "benchmark", "measure performance", "track metrics", "how fast are the gates", "regression check", "baseline", or wants to quantify framework health over time.

Commands

• /benchmark — Full benchmark run (measure + baseline + profile + analyze + report + save)
• /benchmark --quick — Skip profiling (measure + baseline + report only, faster)
• /benchmark --profile-only — Skip baseline comparison, run timing tests only
• /benchmark --save-only — Save current state as a new baseline without running profiling

---

Steps

1. MEASURE — Collect current metrics

Gather the following metrics from live framework state:

Test suite:

python3 ~/.claude/hooks/test_framework.py 2>&1 | tail -10

Extract: total tests, pass count, fail count, pass rate %.

Memory system:

cat ~/.claude/stats-cache.json

Returns: {"ts": <epoch>, "mem_count": <N>}. Also note memory count from search_knowledge("*", top_k=1) result header (total_memories field).

Framework state:

python3 -c "
import json
with open('~/.claude/LIVE_STATE.json') as f:
    s = json.load(f)
print(json.dumps({
    'session_count': s.get('session_count'),
    'framework_version': s.get('framework_version'),
    'feature': s.get('feature'),
}, indent=2))
"

Gate fire rates from today's audit log:

python3 -c "
import json, os, collections
from datetime import date
log = os.path.expanduser(f'~/.claude/hooks/audit/{date.today()}.jsonl')
if not os.path.exists(log):
    print('No audit log for today')
else:
    entries = [json.loads(l) for l in open(log) if l.strip()]
    total = len(entries)
    by_gate = collections.Counter(e.get('gate','?') for e in entries)
    blocked = sum(1 for e in entries if e.get('decision') == 'block')
    warned = sum(1 for e in entries if e.get('decision') == 'warn')
    passed = sum(1 for e in entries if e.get('decision') == 'pass')
    print(json.dumps({
        'total_events': total,
        'passed': passed,
        'warned': warned,
        'blocked': blocked,
        'block_rate_pct': round(blocked/total*100, 1) if total else 0,
        'top_gates': dict(by_gate.most_common(5)),
    }, indent=2))
"