modlens AGENTS.md

Project Overview (for AI Agent)

Goal

Provide the modlens CLI tool that converts image sources (local path or remote URL) into structured text evidence for non-vision LLM workflows.

Scope

The contract: an image the user hands over is read once, and that read leaves text later turns can quote. The image is in the conversation because the user pasted it, dropped a path, or gave a URL.

Do not add:

• a camera, screenshot capture, or hotkeys
• CDP, or holding a browser session
• computer-use (screen plus keyboard and mouse)
• a pixel toolbox (grounding, crop, pixel-diff, reconstructing a UI)
• bounding boxes or confidence scores (dropped from the schema on purpose)

Visual parsing is the only job. Web search and page fetching live in modsearch.

Technical Approach

• Six vision providers behind one interface (src/providers/index.ts). Subprocess providers implement buildInvocation + parseOutput (antigravity-cli, claude-cli, kimi-cli); in-process API providers implement execute (gemini-api, openai, anthropic). antigravity-cli is the zero-config default, and kimi-cli runs only when named, since it spends a subscription.
• Schema-enforced JSON output wherever the backend allows: --json-schema on the Claude and Antigravity CLIs (kimi-cli has no such flag and uses the template), responseJsonSchema on gemini-api, a forced tool call on anthropic. The openai route uses a template-instance prompt (weak gateways echo raw schemas back) plus shape validation that fails loudly.
• Layered config: CLI flags > ~/.modlens/config.json (managed by modlens config init/set/show, 0600, masked rendering) > built-ins. Since 3.17.0 a provider's settings come from one source, whole: the file when it mentions that provider, the bound environment variables (GEMINI_API_KEY, GEMINI_BASE_URL, OPENAI_API_KEY, OPENAI_BASE_URL, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL) when it does not. They used to merge field by field, and an endpoint and a key are one credential, so drawing halves from two places built a pairing that existed in neither. apiKey (file or env) accepts a comma-separated list and rotates after auth, rate-limit, or quota failures. Quota cooldown state lives in ~/.modlens/state.json. MODLENS_MODEL, MODLENS_HARNESS and the proxy conventions are unaffected.
• Vendor knobs pass through, they are not modelled: <provider>.extraBody (or --extra-body) deep-merges a JSON object into the request body of the three API providers, which is how thinking gets turned off. No per-vendor table lives in the code, because the spelling differs per gateway and a wrong guess either 400s or is ignored silently. The fields carrying the image, the prompt, and the schema are reserved (src/util/extraBody.ts).
• Paste recovery across harnesses: modlens recover-paste pulls pasted image bytes out of local session storage (pastes never hit a regular temp file). It supports Claude Code and Pi (JSONL transcripts) and OpenCode (SQLite), detects Codex and defers to its on-disk temp files, and scopes to the harness it runs inside via process ancestry. Exact targeting via --session, else newest-image-timestamp scanning. Storage layouts are each harness's internals, so treat this as best-effort.
• Single responsibility: visual parsing only. Web search and page fetching live in modsearch.