latent-scope AGENTS.md

AGENTS.md — Driving Latent Scope for a user

Cross-provider guidance for AI coding agents (Claude, Codex, Cursor, Copilot, Amp, …) on how to use Latent Scope effectively on a user's behalf — take a raw dataset, turn it into an explorable latent-space map, and show the user the result. Claude Code reads this via .claude/skills/latent-scope/; other agents read this file directly. It is the source of truth; keep provider-specific files (CLAUDE.md) pointing here.

If the user asks you to contribute to Latent Scope (fix a bug, add a feature), see the "Developing" section at the bottom and CLAUDE.md. Most of this file is about using the tool to analyze a user's data.

---

What it is

Latent Scope embeds a dataset, projects it to 2D, clusters and labels it, and serves an interactive map. Pipeline:

ingest → embed → umap → cluster → label → scope → (sprite atlas) → explore

Each step is a CLI (ls-embed, ls-umap, …) that reads/writes files under $LATENT_SCOPE_DATA/<dataset>/. The web UI (ls-serve) drives the same steps and shows the result. Text and image datasets; dense, ColBERT late-interaction, and image (CLIP) embeddings; LanceDB vector storage.

Reach for it when a user has a table/corpus of text or images and wants to see the structure — clusters, outliers, themes, similarity — rather than a single metric.

---

Golden path: run it for a user

On a dev checkout use uv run (resolves .venv). If installed via pip, drop the uv run prefix. Set the data dir and HF cache first:

export LATENT_SCOPE_DATA=~/latent-scope-data     # where datasets live
export HF_HOME=~/hf-cache                         # model/dataset cache

Full pipeline on a text CSV:

uv run ls-ingest   mydata --path /path/to/data.csv --text_column text
uv run ls-embed    mydata text huggingface-jinaai___jina-embeddings-v5-text-nano  # see "Choosing a model"
uv run ls-umap     mydata embedding-001 25 0.1
uv run ls-cluster  mydata umap-001 25 5 0.0 --method hdbscan
uv run ls-scope    mydata embedding-001 umap-001 cluster-001 default "My scope" "description"

ls-cluster auto-writes …-labels-default, so ls-scope … default … works without an LLM. For nicer labels run ls-label with a chat model first and pass that labels id instead of default.