Clawness CLAUDE.md

CLAUDE.md — working on Clawness

Orientation for agents/devs working on this repo. User-facing docs live in README.md; release history in CHANGELOG.md. This file captures architecture, conventions, and the why behind non-obvious decisions.

What this is

A Claude Code plugin that retrieves relevant coding rules and injects them into every prompt via a UserPromptSubmit hook. Pure Python; PyYAML is the only dependency. No ML models, no services, no Docker.

Architecture (request flow)

1. hooks/claude_hook.py (UserPromptSubmit) loads global rules (plugin/clone rules/) + project rules (<project>/.clawness/rules/), retrieves, prints the block to stdout → Claude sees it.
2. Retrieval engine = clawness/core.py: BM25 + TF-IDF fused via RRF, over a concept-expanded token stream (_CONCEPT_GROUPS) + light stemming. Mandatory rules (rules/_mandatory/) always injected; rest ranked + budget-capped. A relevance floor (CLAW_MIN_RELEVANCE, default 0.06, gauged on TF-IDF cosine — not RRF, which is rank-based) drops scattershot matches so signal-less prompts inject few/no ranked rules. Codebase-aware: the hook detects the project stack (detect_stack → scan_project, fresh each prompt) and passes it to Clawness; off-stack language/framework rules (_STACK_DOMAINS minus detected) face a higher floor (CLAW_OFFSTACK_MIN_RELEVANCE, default 0.15) so e.g. a Python repo doesn't surface SQL/React noise, while strong cross-domain matches still pass. Cross-cutting domains (general/meta/workflows/security/testing) are never penalized. science/research are cross-cutting but topically NARROW (_TOPICAL_DOMAINS), so they take a middle floor (CLAW_TOPICAL_MIN_RELEVANCE, default 0.12) between the base and off-stack floors: un-gated, so a researcher in a bare/LaTeX-only directory still gets them, but they must genuinely match. At the base floor (1.3.0) 11 of 30 routine dev prompts surfaced one; the stack filter makes this WORSE, not better, since suppressing off-stack rules frees top-k slots these then fill. That is also why clawness query (no stack) cannot see this class of bug — drive the real hook against a project fixture instead. cfd/julia/fortran/matlab/r are the mirror case (_NARROW_STACK_DOMAINS, 1.6.0): stack-gated AND vocabulary-colliding, so off-stack they take a fourth floor ABOVE the ordinary one (CLAW_NARROW_MIN_RELEVANCE, default 0.22). Their core words are ordinary dev words — measured against the real hook in a Python repo, "the solver is not converging, fix the residual bug" scored CFD-CONVERGE-001 at 0.190 and "vectorize this dataframe loop" pulled in MATLAB (0.193) and R (0.163), all clearing the 0.15 off-stack floor. Routine dev prompts top out at 0.193; an explicit ask ("which turbulence model for this openfoam case") starts at 0.264, so 0.22 sits in the gap. The high bar costs them nothing where they matter — in their OWN project they are on-stack and the floor never applies. Don't fold them back into the plain off-stack tier: unlike sql/docker (a Python service really does talk to Postgres) there is no such thing as needing Fortran conventions while writing TypeScript. Note _floor_for must test the narrow set BEFORE returning the off-stack floor — these domains are in _STACK_DOMAINS too, so the reverse order makes the tier dead code. Passing no stack (CLI/eval) disables the penalty, so eval is unaffected. ~4ms/prompt + ~10ms scan (still ~3% of the ~400ms hook, which is dominated by interpreter startup). The stack scan walks the tree once; it does not glob per pattern (1.18.0). Path.glob is NOT recursive, so for ten versions every *.py/*.css/*.sql/*.tex detector only ever saw the repo ROOT — a project keeping its stylesheets in src/styles/ never triggered css, and the manifest detectors masked it well enough that nobody noticed. rglob is not the fix: **/*.py measures ~121ms against ~0.4ms shallow, on a path that re-runs uncached every prompt. So _walk_project lists the tree ONCE, bounded at depth 4 / 4000 entries and skipping init.SCAN_SKIP_DIRS (one source of truth: a directory not worth searching for a hostname is not worth searching for a stack — the set lives in init and guard imports it, NOT the reverse, because importing guard here would put its ~13ms of regex compilation on every prompt), and _pattern_hits matches every pattern against that one listing — by BASENAME at any depth, or by trailing path segments for a pattern that carries a /. Matching is indexed, not fnmatch.filter per pattern: filtering all ~50 patterns over the walked names measured ~9ms, more than the walk itself, so bare *.ext patterns (most of them) resolve through an extension dict and only the rest compile a cached matcher. A leading ./ opts a detector back OUT of recursion (./main.py, ./app.py, ./DESCRIPTION): "a main.py at the top of the project" is a fair guess at a web app, "somewhere in the tree" is not — unanchored, this repo's own tests/fixtures/vuln/app.py dragged fastapi in. Expect a detector to need anchoring whenever its filename is common as a nested file. coverage.detect_uncovered deliberately stays a shallow root glob — it drives "Clawness has no rules for your stack", which must not fire on one vendored .rb. Session-aware re-injection (clawness/session_state.py): the mandatory block (identical every turn) renders in full only on prompt 1 and every CLAW_FULL_EVERY-th prompt after (default 5); other turns get a one-line id list — the rules stay just as binding, only their re-statement shrinks. State is a per-session JSON file in the OS temp dir (never the project), fails toward a full render on any error. Project memory does NOT ride this cadence — see below.
3. Project memory (<project>/.clawness/memory.md, logic in clawness/memory.py): a per-codebase lessons log, appended after the rules block. Retrieved, not dumped (since 1.2.0): parse_memory splits the file into ## Always entries (pinned, always injected, capped by CLAW_MEMORY_PIN_BUDGET) and ## Lessons entries, which rank_lessons ranks against the prompt — same BM25 + TF-IDF + RRF primitives as the rules, so a 200-entry log still costs a flat handful of lines. HTML comments and headings are stripped: they're for the human editing the file and cost ~107 tokens/turn on an otherwise empty log, which is what prompted the rework. Three deliberate choices:
   • Memory ranks in its OWN pass, never merged into Clawness._ranked_rules. Lessons can't displace rules from top_k, rules can't displace lessons, and rank_ids stays rule-only so tests/ground_truth.json and the CI eval floors are immune to whatever a user writes in their memory file.
   • memory.py has its own stopword list. Across 113 rules, IDF flattens "this"/"the"/"needs" on its own; across a 4-40 entry log those words look discriminating, and without the filter "rename THIS variable" matched "BUILDKIT=1 on THIS machine" above the floor. Don't remove it thinking IDF covers it.
   • CLAW_MEMORY_MIN_RELEVANCE defaults to 0.20, not the rules' 0.06. Small-corpus cosines run hot; measured, genuine hits score 0.44-0.70 and incidental overlap 0.07-0.09, so 0.20 sits in the gap. Because the block is already prompt-specific and ~3 lines, it ships every turn rather than being abbreviated. memory_changed (session_state) now drives force_recent instead of a cadence: the newest entries show on the session's first prompt and on any turn after the file changed, so a lesson written mid-session is never invisible. Memory (and the few fixed suggested-action lines) sit outside CLAW_BUDGET by design — counting them in would make rule selection vary with memory length; total injection ≈ CLAW_BUDGET + CLAW_MEMORY_BUDGET + a few fixed lines. ENF-MEM-001 (mandatory) is the single rule telling Claude to maintain the file and carries the numeric contract (one line, <=120 chars, max 3 pinned, prune past 40); it absorbed the near-duplicate ranked WF-LESSONS-001, which is gone. The file is auto-created on first session by hooks/memory_init.py (SessionStart) — gated to git work trees, opt-out CLAW_NO_MEMORY; it injects a note (like git_check) so Claude announces the file to the user, since hooks can't prompt directly. That hook has a second, independently gated concern (1.8.0): offering the .gitignore block for .clawness/. Same consent shape — it asks, never edits. memory.md and rules/ are meant to be committed; handoff.md, handoffs/ and the ledgers are per-machine. The block is an allowlist (.clawness/* plus ! for the two shared paths) so a ledger added in a future version is ignored by default, and the trailing /* is load-bearing — ignoring the bare directory stops git descending and the negations silently do nothing (tests/test_memory.py pins this by applying the block for real and asking git check-ignore). Coverage is asked of git (check-ignore), not of .gitignore's text, so a global or wholesale rule counts and is left alone. It needs its own ledger (.clawness/gitignore.json) because, unlike creating memory.md, a declined offer isn't self-limiting; checked LAST, as everywhere else. The two halves don't gate each other: a project that predates 1.8.0 has a memory.md already and still needs the ignore rule.
4. Context-pressure watch (clawness/context_watch.py, called from claude_hook): reads the session's own transcript (transcript_path in the hook payload; falls back to reconstructing <config>/projects/<slugified-cwd>/<session_id>.jsonl) and warns the user before the window fills and quality degrades. Rides the existing UserPromptSubmit hook rather than adding one — it's a file tail plus arithmetic (~0.5ms), not worth another process spawn per prompt.
   • Context size is read, not estimated. The last assistant entry's input_tokens + cache_creation_input_tokens + cache_read_input_tokens IS the prompt that was just sent. Only the last 256KB of the file is read (a transcript reaches several MB; a 6MB tail costs ~0.7ms), walking backwards to the newest usage record.
   • The window can't be read from the transcript — a 1M session records the same claude-opus-5 model id as a 200k one. infer_limit therefore goes CLAW_CONTEXT_LIMIT → the [1m] marker on model in settings(.local).json → observed-usage tier bump. Don't drop the settings check: without it a 1M session false-alarms all through 140k-200k, which is exactly how users learn to ignore the warning.
   • Levels: warn (70%, brief mention), urgent (85%, recommend a fresh session and offer a handoff + memory write), and surge — a single turn adding >=12% of the window with <=5 turns of headroom left, so a session filling fast is flagged while there's still room to act. Below MIN_TOKENS_TO_REPORT (20k) it never speaks.
   • Each level alerts at most once per session (should_alert_context); escalation warn→urgent passes, repeats don't. The condition stays true once reached, so without dedup it would fire every prompt for the rest of the session.
   • Fails silent on every path, opt-out CLAW_NO_CONTEXT_WATCH.
5. Session handoff (clawness/handoff.py + hooks/handoff_check.py, SessionStart): the other half of the context watch. At ~85% full it offers to write <project>/.clawness/handoff.md; the SessionStart hook injects that file when the next session in the project starts, so the user never has to remember it exists or know its path. WF-HANDOFF-001 (ranked) tells Claude where and how to write one.
   • handoff.md and memory.md are different things and must not be merged. memory.md accumulates durable lessons and is committed/shared; handoff.md is one transient "here's where I was", overwritten each time and personal (gitignore it).
   • The note injects the handoff's CONTENT, not a pointer. A pointer costs the next session a tool call and depends on Claude choosing to follow it — the whole point is that the user shouldn't have to shepherd the pickup.
   • The file's existence IS the state — a handoff at that path hasn't been picked up. There is deliberately no age cutoff or done-flag: an old handoff nobody archived is still outstanding. archive_handoff moves it to .clawness/handoffs/done/<timestamp>.md when superseded (a new one is written, or the user says it's finished), which clears the live slot and keeps history. Nothing is ever deleted; an over-eager archive then costs nothing. Age IS shown in the note, but only as information — it never branches the instruction.
   • "Carry on" means START, not summarize-and-wait (1.6.0). The instruction used to end "then wait for them. Don't start the work unless they ask", which threw away the reason the handoff exists — the user wrote it so the next session wouldn't need an interview. It is now conditional, and must stay conditional: SessionStart fires before the user's first message, so the note cannot know whether they will say "carry on" or open a fresh task, and it has to carry both branches. What makes the continue branch safe is the template's ## Open questions section — the note bounds questions to what is listed there, so "don't ask" can't mean "guess". If you ever drop that section, restore the interview.
   • The session name is a SUGGESTION, and is now opt-in, default OFF (1.9.0; gated 1.11.0). An unnamed session is titled from the user's first message, so every pickup reads "carry on" in their history — the one phrase all pickups share. Claude Code has a built-in /rename [name] (alias /name; bare, it generates a kebab name from the conversation), but a slash command can only be TYPED: no hook can rename a session and neither can Claude. suggest_session_name derives a name from the handoff's # heading and, when CLAW_HANDOFF_SUGGEST_NAME is set, the note surfaces the one-liner once, on the pickup branch only — the heading is the wrong name for a session the user opened on something else. Surfacing it on every pickup proved more nagging than it was worth, so the clause is silent by default; only the note mention is gated, suggest_session_name itself stays live and tested. It returns "" (and the note says nothing) when the heading has no letters after cleaning, which is exactly the template's default # Handoff — {date}: a suggestion the user has to read and reject costs more than silence. Don't "improve" this by writing name/nameSource into <config>/sessions/<pid>.json — that file is the CLI's own live state, in an undocumented shape, and it may well hold it in memory anyway.
   • Truncation keeps the head (budget CLAW_HANDOFF_BUDGET, default 2000) — opposite of the lessons log, because a handoff's summary and state are written at the top. ## Open questions is last and therefore the first thing truncated away; that's the right trade (it usually says "none") and tests/test_handoff.py pins it rather than leaving it undefined. Opt-out CLAW_NO_HANDOFF.
6. Framework-version awareness (VERSION_WATCH_* in clawness/init.py, surfaced by stack_detect): scan_project returns a versions dict alongside domains, and the SessionStart note reads "Next.js 14.2, React 18.3" rather than bare labels.
   • Parsed at SessionStart ONLY. detect_stack (per-prompt) still reads domains and ignores versions, so the hot path is untouched. scan_project re-runs on every prompt uncached — don't move version parsing onto that path without a TTL cache first (the guard_provenance_cache.json pattern).
   • Unparseable means omitted, never guessed. *, latest, a git URL or a workspace protocol yield "" and the framework falls back to its bare label. A wrong version is worse than none: it gets acted on.
   • The watch list is deliberately short — only frameworks whose majors change the code you write (App Router vs Pages, Pydantic v1 vs v2, SQLAlchemy 1.4 vs 2.0). It is not an inventory; the manifest is right there. GEN-INSTALLED-VER-001 is the durable half, and it is distinct from GEN-DEPS-001/ENF-SEC-005, which are about choosing a version for a NEW dependency rather than matching an existing one. 6b. Corpus staleness (clawness/staleness.py, surfaced by stack_detect's check_staleness; remedy in skills/refresh/SKILL.md): a rule may carry applies_to/verified/sources recording the versions it was established against, and the note fires when the project declares one past that range.
   • The relevance floor cannot catch this. It detects unfamiliar vocabulary; a major bump keeps the words ("route", "cache", "app router") and changes their meaning, so a 14-era rule scores like an ordinary match on a v17 prompt. Measured on the live corpus: NX-CACHE-001 at 0.112, NX-ROUTE-001 at 0.121.
   • Stamps are per RULE, never per domain. A domain range is the union of its rules' ranges — structurally the widest claim available — and too-wide fails silent while too-narrow produces a visible false alarm that gets corrected. Don't "simplify" this by stamping folders.
   • Only a verified stamp arms the warning (is_armed): applies_to without verified and sources is asserted, not established, and stays silent. The feature therefore ships doing nothing until real review has happened — that is the honest behaviour, not a shortcoming. Establishing a range is an OUTPUT of review; there is no way to derive it (git dates are a weak proxy, rule text names an API not a version).
   • Grammar is an inclusive "13-15" / "15" / "1.4-2.0", one or two numeric components per bound — the shape _clean_version produces. Ceilings pad with a sentinel ("15" covers 15.9) because padding with 0 would false-alarm on 15.1. Open-ended ("13-") is deliberately inexpressible: "and everything after" is the claim nobody has evidence for. Below-floor mismatches are silent — the corpus is written forwards, so warning there fires on every un-upgraded project.
   • The join key is the DETECTOR's label ("Next.js"), since that is how scan_project keys versions. A typo'd label never matches, so the check goes silently inert while looking configured — which is why clawness lint validates membership in WATCHED_LABELS mechanically. That is the check to watch fail first (TST-FAILFIRST-001).
   • The note orients; it does not commission work, and its text is a tested artifact (TestNoteText). An earlier attempt had Clawness author rule files from the automatic path and it wrote heaps of them, eating the session — the same failure as 1.7.0's CLAUDE.md remedy, and for the same reason: a SessionStart note fires before the user has said what they came for. So the note names /clawness:refresh and starts nothing, permits only a passively triggered one-line append to .clawness/memory.md (SESSION_BACKSTOP, a judgment call — the real bound is "if you happen to establish… while doing the user's work"), and never names .clawness/rules/.
   • Reads stamps from global AND project rules, project winning by id, so rules /clawness:refresh generates are stale-checked by the same mechanism when the project moves again. Without that they'd be the one class that never can be.
   • The ledger keys on the fact, not a date (.clawness/staleness.json stores label → detected version): once per mismatch, re-arming when the version moves. A "checked today" flag was rejected — it goes silent for the rest of the day if the user upgrades at 2pm having been checked at 9am, and re-asks forever once declined. Same shape as claude_md_check's size ledger. Revisit only if a future version asks npm/PyPI what the current major is (network I/O, where a TTL cache becomes correct). unasked is called LAST; opt-out CLAW_NO_STALENESS_NOTE; fails silent on every path.
   • Don't auto-suppress stale rules. A rule that's 80% right beats silence, provided Claude knows to check — which is what the note buys.
   • Known limit: covers only the ~14 VERSION_WATCH_* packages, detects version drift only (not a framework abandoned outright, nor a rule wrong when written).
   • Four whole domains are structurally unstampable, found while stamping for 1.9.0 — don't rediscover them. python has no join label at all: the watch list is frameworks, not the interpreter, so a "Python" key fails lint and there is nothing else a PY-* rule can key on. Adding one would mean detecting the interpreter version, which _python_version (a dependency reader) does not do. And the fastapi-labelled rules are deliberately left bare because FastAPI ships 0.x: _clean_version yields two components, so the effective major is the minor, which moves every few weeks — any ceiling false-alarms almost immediately. The version-sensitive claims in that domain hang off Pydantic and SQLAlchemy instead, which have real majors. typescript and css are unstampable for the opposite reason — the label has a fine major, the CLAIM doesn't. "Enable strict, prefer unknown to any", "?? not ||", "Flexbox for one dimension, Grid for two" were true before TypeScript 7 and will be true after 8, so a ceiling on them buys exactly one guaranteed false alarm per major. (css has no Tailwind rule at all — the Tailwind label is in VERSION_WATCH_JS for the stack note's benefit, not because anything keys on it. Check before assuming a label implies corpus.) The general shape: a stamp is only worth writing where the major changes the claim — the label having a major is necessary, not sufficient.
   • A review can find the rule WRONG, not just unstamped, and that is the point. Two of the 1.9.0 domains turned up rules teaching a hazard that had reversed: SCI-ARRAY-001 on pandas views (3.0's Copy-on-Write inverted it — chained assignment now silently no-ops where it used to mutate-and-warn) and CAP-WEBVIEW-001 on the Status Bar plugin (inert under Capacitor 8's unconditional edge-to-edge). Prefer rewriting the rule to carry BOTH eras over capping it at the old major: a ceiling makes users on the old version see a note about a rule that is still correct for them, while the rewrite serves everyone and the stamp then records the range you actually checked.