chronicle: Instructions file for Claude Code

CLAUDE.md — chronicle

Context for agents and contributors. The deployment-specific half — host, schedule, stacks, open operational items — lives in CLAUDE.local.md (gitignored) on a machine that runs chronicle. User-facing docs: README.md.

What this is

A self-hosted personal event store and retrieval layer. It ingests a chat archive and the activity streams its owner already self-hosts into one timeline, groups events into segments before indexing, and serves the result to an assistant over MCP.

The one idea everything follows from

The reference archive was measured live, not assumed:

messages	681,331 across 487 chats, 457 senders
span	2018-12-30 → present (7.6 years, ~245/day)
under 20 chars	65.0% (442,954)
under 60 chars	94.0%
over 200 chars	1.5% (10,484)
with reply_to_id	7.9%
voice + video notes	18,638 · photos 12,100 · documents 2,113

The reference deployment's Telegram exporter embeds every message into Qdrant, so ~442,000 of its vectors represent ок, ага, +1, да, 😂. They crowd every ANN neighbourhood and bury the 1.5% that carry a proposition.

The core move is aggregation before indexing. Events group into segments by a per-thread fitted time gap: 681k units → ~50k. The index gets ~11× smaller and retrieval gets better simultaneously.

Evidence (SeCom, ICLR 2025, retrieval quality by memory unit, at ~30 tokens/turn where ours are 5–10):

segment-level   71.57   <- what this builds
turn-level      65.58
session-level   63.16
summaries       53.87-56.25   <- WORST. Never build a summary pyramid.

Full reasoning and citations: docs/RESEARCH.md (16k words).

Commands

make test                 # 195 unit tests; bootstraps .venv, no DB or models
make smoke                # migrations + every SQL function, throwaway DB
make doctor               # validate sources BEFORE ingesting  ← always first
make ingest               # sources -> event -> segment -> embedding
./scripts/nightly.sh      # ON THE BOX: what cron runs — all stages, tier 2
make eval-init && make eval   # chronicle vs ripgrep on real questions
make resegment CAP=15 THREADS="$(make -s eval-threads)"   # a cap sweep's rebuild
make purge-excluded       # what the filters now exclude but already indexed
make purge-excluded APPLY=1   # ...and delete it, in one transaction
make redact-secrets       # count stored events holding a secret (APPLY=1 rewrites)