personal-knowledge: Skill for Claude Code

eval-memory — memory system health & efficacy runner

You are the runner. This skill is a procedure you execute live, mostly quiet, pausing to converse only where the user's judgment matters (the verdict and the dashboard gap). Default runs are read-only and safe to run unattended.

The one idea that makes this non-trivial

Every autolearn signal is self-referential: co-access feeds re-rank, which shapes what co-surfaces, which feeds co-access. So "the numbers are growing" proves activity, not improvement — it can be a self-confirming loop. The only way to claim "it's working" is a ground-truth judge external to the signal loop. That judge is you, ruling blind on whether re-ranked retrieval beats raw retrieval. Protect that blindness or the whole eval is theater.

Config (read, don't hardcode elsewhere)

• Server base URL: http://127.0.0.1:7345 (from MEMORY_HTTP_HOST/MEMORY_HTTP_PORT in .env).
• Repo data dir: data/ (gitignored). Key files: search-history.json, auto-signals.json, .autolearn-state.json, autolearn.config.json.
• Frozen probe set: eval/probe-set.json (local, gitignored — copy from eval/probe-set.example.json and author your own; the controlled ruler — see Q4 below).
• Outputs: eval/ledger.jsonl (trend, append-only) and eval/reports/<YYYY-MM-DD>.md (narrative).

Procedure

Phase 0 — Setup (quiet)

1. Read eval/probe-set.json. If missing, stop and tell the user to seed it (see "Seeding" below).
2. GET /health/detail. Record autolearn_level, totals, last_flip_at, last_backup_*, and whether learning_stats is present.
3. Read the last few rows of eval/ledger.jsonl (if any) for trend comparison.

Phase 1 — Health (quiet batch, Q9/B)

Collect, don't narrate each item. Three layers:

• Plumbing: /health reachable; /health/detail.autolearn_level == expected (activation per current .env); totals sane vs ledger; learning_stats present iff level==activation.
• Freshness (high-value — catches a silently-dead 60s flush timer): is auto-signals.json and search-history.json mtime / newest-entry ts recent relative to known activity? A stale file at activation level is a red flag — learning has quietly stopped while everything looks fine.
• Corpus hygiene: run analyze_subject on the top live subjects (your most active projects and the memory system) and/or scan totals for: orphans, untagged, state past expected_until, decisions-without-outcomes, contradictions, and obvious noise (test/tombstone/probe records). Produce a compact health summary with PASS/WARN/FAIL per check.