latent-scope: Instructions file for Claude Code

CLAUDE.md — Latent Scope

Guidance for AI agents (Claude Code and others) working on this codebase. If you are an agent asked to "run the pipeline" or "show me the results", jump to Agent quickstart.

Using Latent Scope for a user (running the pipeline, choosing a model, serving results) → the cross-provider runbook is AGENTS.md (Claude Code also loads it via .claude/skills/latent-scope/). This file (CLAUDE.md) focuses on the codebase itself — layout, design decisions, and contributing. When the two overlap, AGENTS.md is authoritative for usage.

---

What Latent Scope does

Latent Scope is a Python + React tool for embedding, projecting, clustering, labeling, and exploring datasets through the lens of their latent space. The backend is a Flask server; the frontend is a React/Vite SPA. Pipeline steps run as CLI subprocesses (ls-embed, ls-umap, …) that can be driven from the web UI or scripted directly.

ingest → embed → umap → cluster → label → scope → (sprite atlas) → explore

Capabilities (what an agent can offer a user)

• Text and image datasets. Ingest CSV/Parquet/JSON/JSONL/XLSX, a pandas DataFrame, or a directory of images (ls-ingest <ds> --path <dir> reads bytes + filename/date/size_kb columns). Image columns (HF {bytes,path} dicts, raw bytes, or URLs) are auto-detected and embeddable (issue #87); local file paths in a table are not.
• Dense embeddings from many providers (sentence-transformers/HF, OpenAI, Cohere, Voyage, Mistral, Together, and any OpenAI-compatible endpoint).
• ColBERT late-interaction (multi-vector) embeddings via pylate — per-token vectors stored fp16, searched with MaxSim (issue #64). See examples/colbert_quickstart/.
• Token maps: for late-interaction embeddings, an alternative granularity that maps one point per token (not per document) while the table shows parent documents with the selected token highlighted in context. Pipeline: ls-tokenize → ls-umap --granularity tokens → ls-cluster (token-frequency default labels) → optional ls-sae --granularity tokens --checkpoint <dir> (token-level SAE features) → ls-scope. See docs/token-maps.md.
• LanceDB vector storage (replaced HDF5). Embeddings live in a per-dataset LanceDB table; old HDF5 embeddings are migrated on demand.
• UMAP projection + clustering, with LLM cluster labeling. Four clustering methods: ls-cluster --method {evoc,hdbscan,kmeans,gmm} with a --cluster_on {umap,embedding} input choice (issue #41). For kmeans/gmm the samples positional is the number of clusters. See docs/clustering.md.
• GPU acceleration (optional): cuML-accelerated UMAP, HDBSCAN, and KMeans on NVIDIA GPUs with graceful CPU fallback, controlled by the LATENT_SCOPE_DEVICE env var (cpu|cuda|auto, default auto). Install pip install "latentscope[gpu]" --extra-index-url=https://pypi.nvidia.com. EVoC and GMM stay on CPU. See docs/gpu-acceleration.md (issue #63).
• Named experiments + gallery: umap/cluster runs take optional --name/--description (stored in their metadata JSON, editable in Setup); the Setup UI shows a browsable thumbnail gallery instead of a bare id list.
• Explore UI: GPU scatterplot with hover/select, density heatmap, cluster outlines, similarity + late-interaction search, filtering, tagging, and color-by any numeric/categorical column (issue #131). The Compare page shows two scopes side by side with a shared lasso/brush selection (issue #132).
• Image map (sprite atlas): for image datasets the map is a continuous level-of-detail — heatmap when zoomed out, a tiled representative-image pyramid as you zoom in, then individual points on top for hovering. Built by the optional post-scope ls-sprite-atlas step (issue #24).