membot CLAUDE.md

CLAUDE.md — membot

Guidance for Claude Code when working in this repo. Pair with docs/plan.md (the source-of-truth design doc).

What this project is

membot is a standalone Bun CLI + MCP server (Bun package membot, binary membot) that gives AI agents a persistent, versioned, searchable context store. Files (markdown, PDF, DOCX, HTML, URLs) are ingested, converted to markdown, chunked, embedded locally with @huggingface/transformers (WASM, 384-dim Xenova/bge-small-en-v1.5), and indexed in DuckDB with hybrid search (vector + BM25). Every agent-visible artifact is a row in files, addressed by a virtual logical_path — there is no on-disk tree of stored content.

Reference project (origin):

• botholomew — the chunker, embedder, markdown-converter, and hybrid search were originally embedded here. The earlier mcpx-based fetcher has been replaced; today's fetcher is a per-service downloader registry (see "Architecture at a glance" below).

Hard constraints

• Bun-only. No Node-only deps. bun build --compile produces standalone binaries; the runtime must not require Bun installed.
• Local embeddings only. @huggingface/transformers WASM, Xenova/bge-small-en-v1.5, 384-dim. Never reach for cloud embedding APIs (OpenAI/Voyage/Cohere/Anthropic embeddings) even if a reference project uses them.
• DuckDB is the only store. Content AND original bytes live in rows (files.content, blobs.bytes), not in a filesystem tree. ~/.membot/index.duckdb holds everything except cached model weights. The DB will get large — that's accepted.
• Append-only versioning. Every ingest, refresh that finds new bytes, write, or rename creates a new (logical_path, version_id) row. version_id is a TIMESTAMP (ms precision). Default queries flow through current_files / current_chunks views. Delete = tombstone, not a row removal.
• MCP defaults to current. Every MCP tool acts on the latest non-tombstoned version unless version is passed explicitly.
• Per-service downloaders + persisted provenance. Remote URLs are dispatched to a source-plugin registry (src/ingest/sources/): GitHub via REST API + PAT, Linear via GraphQL + personal API key. There is no generic-web catch-all and no Google plugin — arbitrary URLs (and Google Docs/Sheets/Slides URLs) are rejected with a clear HelpfulError. Each row persists (downloader, downloader_args) so refresh replays the exact same downloader against the same URL — deterministic, no LLM, no agent loop.
• Fetches are non-interactive. membot add and membot refresh never prompt or open a browser. Auth failures throw HelpfulError with a concrete next step (membot config set downloaders.<svc>.api_key for token services).
• No Playwright, no Chromium, no Google ingest. Membot doesn't launch a browser, embed one, or shell out to a third-party CLI. Google Docs/Sheets/Slides aren't a first-class source — the OAuth scope tax to get Drive access (either cloud-platform to gcloud or a manual GCP-project setup) was disproportionate. Workaround for users: export the Drive file as .docx / .xlsx / .pdf and membot add <path>.
• Native conversion first, LLM fallback for messy/binary input. unpdf, mammoth, turndown handle the common cases. Claude vision captions images; Claude markdown-converter is the last-resort fallback. Missing ANTHROPIC_API_KEY is not a hard error — the pipeline degrades to deterministic surrogates.
• Textual surrogate is the universal interface. Every artifact (markdown, PDF, image, audio, anything) produces a markdown body that flows through chunking + embedding + FTS. Original bytes live in blobs and are reachable via membot_read bytes=true. Search has zero special cases for binary content.
• Always describe. files.description is generated for every ingested file, including plain markdown. The string <logical_path>\n<description>\n<heading-breadcrumb>\n\n<chunk_content> is what gets embedded and FTS-indexed (stored as chunks.search_text); chunks.chunk_content keeps the raw body for clean snippet rendering. The description is capped (~240 chars) and the breadcrumb line is present only for heading-scoped markdown chunks, so the whole search_text fits bge-small's 512-token window.
• Size chunks to the model window, pool the way the model was trained. Chunk sizing (chunker.target_chars/max_chars, default 1400/1800) is budgeted so search_text fits bge-small's 512-token limit — oversize chunks silently embed only their prefix. BGE-v1.5 uses CLS pooling (resolvePooling in embedder.ts), not mean. Markdown is chunked at heading boundaries (fence-safe) with a per-chunk breadcrumb (chunker.markdown_aware, default on). Any change to pooling, chunk sizing, or the search_text shape MUST bump EMBEDDING_REVISION (constants.ts) with a history line; existing stores clear the resulting search-time warning by running membot reindex --embeddings.
• membot_add accepts directories and globs. Single arg, polymorphic: file path, directory (recursive walk, symlinks followed via realpath dedupe), glob (docs/**/*.md), URL, or inline:<text>. Each matched entry becomes its own version under its own logical_path; partial failures are reported per-entry, not all-or-nothing.
• CLI auto-renders for the environment. TTY → spinners, progress bars, ANSI colors. Piped/--json/CI=true/NO_COLOR → JSON to stdout, structured logs to stderr, no ANSI bytes. One code path; src/output/tty.ts is the single source of truth for which mode is active.
• All errors are HelpfulError. Bare throw new Error(...) is forbidden. HelpfulError requires a non-empty hint (statically and at runtime); the hint must name the next action concretely. The same hint string lands in front of both humans (CLI stderr) and LLMs (MCP structuredContent.error.hint and the rendered text content).
• No any. Both implicit and explicit any are banned (TS strict: true + biome noExplicitAny: error). For untyped third-party APIs, declare a local interface that captures the methods you actually use and cast the import once at the boundary; sprinkle unknown + type guards everywhere else.
• Every method gets a docstring. Every exported (and most internal) function, method, or class member must have a JSDoc-style comment that explains what it does — preferably also why when the rationale isn't obvious. One short line is fine for trivial wrappers; multi-line comments are appropriate for orchestration paths or anything with a non-obvious contract. Don't restate the signature; explain the intent and the contract.
• Tests are written alongside code, not bolted on. Every new module ships with unit tests covering the happy path, the error path, and the edge cases (empty input, malformed input, boundary conditions). DB-touching code uses real ephemeral DuckDB files, not mocks. Error types are tested for both their invariants and their rendering.
• Migrations are logged. Every migration applied at startup writes an info line so users can see what changed when they upgrade.
• User-facing changes bump package.json. Any change that ships behavior to users (new flag, new command, fixed bug they could observe, output-format change) must increment version in package.json in the same PR. The auto-release workflow only fires when the version changes — no bump means no release and no binaries. Internal-only refactors, comment edits, and test changes don't need a bump.