lethe: Instructions file for Codex

lethe — agent guide

Self-improving memory store for LLM agents. BM25 + dense vector hybrid retrieval with cross-encoder reranking and clustered retrieval-induced forgetting (RIF). Indexes agent transcripts (Claude Code / Codex) directly — no capture hooks, no summarization. Ships as a single Rust binary (lethe) plus PyPI / npm bindings.

stack

• Rust workspace (crates/) — production. Toolchain pinned via rust-toolchain.toml.
  • lethe-core — library: tokenize, bm25, faiss_flat, rrf, dedup, rif, kmeans, encoders (ONNX via ort), DuckDB persistence, npz reader, memory_store, union_store, markdown_store, transcript_store.
  • lethe-cli — lethe binary (clap; embeds the TUI).
  • lethe-tui — ratatui library called from the CLI on no-arg invocation.
  • lethe-py — PyO3 binding → PyPI lethe-memory.
  • lethe-node — napi-rs binding → npm lethe.
  • lethe-benchmark — internal parity bench helper (publish = false).
• DuckDB for entry metadata + embedding BLOBs (single source of truth).
• ONNX Runtime (via ort) for the bi-encoder + cross-encoder. Default models: Xenova/all-MiniLM-L6-v2, Xenova/ms-marco-MiniLM-L-6-v2.
• Python research_playground/lethe_reference/ kept for the research-journey trail: produces the benchmark numbers cited below. Installed as lethe-memory-legacy in dev venvs; not published.
• Datasets (tmp_data/, gitignored): LongMemEval (S), NFCorpus.

commands

# Rust workspace
cargo build --workspace --release        # → target/release/lethe
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check

# Pre-commit / pre-push hooks (replaces the deleted ci.yml)
cargo install --git https://github.com/j178/prek
prek install                             # writes .git/hooks/pre-commit + pre-push
prek run --all-files                     # one-off run

# Python reference impl (research trail; library only)
uv pip install -e research_playground/lethe_reference/
cd research_playground/lethe_reference && uv run pytest tests/ -v     # 148 + 8 PyO3 parity = 156
uv run python research_playground/baseline/run.py

# Python ↔ Rust parity bench
uv run python research_playground/rust_migration/prepare.py
uv run python research_playground/rust_migration/longmemeval.py --compare
uv run python research_playground/rust_migration/components.py --compare
uv run python research_playground/rust_migration/latency.py --compare

# CLI surface (the `lethe` binary)
lethe                                    # no args → TUI (in a terminal)
lethe index                              # index this project's transcripts
lethe index --all                        # reindex every registered project
lethe search "query" --top-k 5           # reindexes changed transcripts first
lethe search "query" --all --top-k 5     # all registered projects (per-project read-only)
lethe projects list|add|remove|prune
lethe expand <chunk-id> [<chunk-id> ...]
lethe status