workspace-docs-mcp AGENTS.md

AGENTS.md

Instructions for AI agents working in this repository.

Project overview

This is journal-rag, an MCP server + CLI that provides hybrid BM25 + vector semantic search over markdown journal files. It's designed to be installed once and used across multiple consuming repos, each with their own journal-rag.config.json.

Architecture

src/
  server.ts     — Lightweight MCP stdio proxy entry point
  daemon.ts     — Shared per-user IPC daemon entry point
  daemon-client.ts / daemon-protocol.ts — Daemon startup, transport, and protocol
  workspace-runtime.ts — Deduplicated workspace state and tool operations
  logger.ts     — Cross-platform daily JSONL diagnostics
  indexing-lock.ts — Cross-process vector-build lock
  cli.ts        — CLI entry point (journal search/list/get/regex/index)
  index.ts      — Markdown file discovery, heading-based chunking, BM25 index cache
  embeddings.ts — Local vector embeddings via @huggingface/transformers, vector cache
  search.ts     — BM25, substring, regex, and hybrid (RRF) search implementations
  config.ts     — Config file discovery and parsing
  types.ts      — Shared TypeScript interfaces

Key design decisions

• Hybrid retrieval: search_journal fuses BM25 keyword scores with vector cosine similarity using Reciprocal Rank Fusion (RRF, k=60). This avoids score normalization issues.
• Shared daemon: Every editor needs its own lightweight stdio MCP proxy, but all proxies connect to one per-user daemon over a Windows named pipe or Unix socket. No OS service installation is required.
• Workspace deduplication: Daemon runtimes are keyed by the canonical path to journal-rag.config.json, so Cursor and ChatGPT share one runtime when they open the same workspace.
• Shared local embeddings: The daemon loads onnx-community/Qwen3-Embedding-0.6B-ONNX through @huggingface/transformers. Model loads and inference calls are serialized; model promises are deduplicated by model ID.
• Platform acceleration: Use DirectML on Windows, Core ML on macOS, and CPU elsewhere. DirectML must use enableMemPattern: false and sequential execution.
• Conservative indexing: Embed one journal chunk per batch. This avoids DirectML device hangs on 6 GB GPUs. A proper-lockfile lock serializes vector builds across processes and recovers stale locks.
• Incremental vector cache: Stored at .journal-rag/vectors.json alongside the BM25 cache. Only new or missing chunks are embedded, deleted chunks are pruned, and partial results are checkpointed every 100 chunks.
• Graceful degradation: If the vector index fails to build (e.g., model download issue), the server falls back to BM25-only search without crashing.
• Heading-based chunking: Markdown files are split at ## and ### boundaries. Each chunk carries its heading path for citation.