perplexity-webui-scraper: Instructions file for Codex

AGENTS.md — project field guide

Read this before changing code. It explains what this project actually does, which boundaries protect it, and how to verify a change without breaking user-facing contracts.

What this project is

perplexity-webui-scraper is a typed Python library that talks to Perplexity's web interface through undocumented internal endpoints, authenticating with the __Secure-next-auth.session-token cookie. It is not an official API client and does not scrape HTML pages. Upstream behavior may change without notice; treat endpoints, payloads, and SSE formats as fragile integration boundaries that need observable behavior and tests.

The package offers four entry points over one shared core:

1. Python library: Perplexity creates Conversation objects. Conversation.ask() owns thread state, uploads attachments, supports streaming, and exposes answers and sources.
2. CLI: one Typer command (perplexity-webui-scraper) delegates to chat, email authentication, API, and MCP commands. Optional extras keep optional dependencies out of the base installation.
3. OpenAI-compatible REST API: FastAPI accepts a Perplexity token on each Authorization: Bearer ... request, converts messages/configuration, and returns JSON or SSE. Compatibility is intentionally partial: unsupported fields may be ignored and token usage is reported as zero.
4. MCP server: registers one tool per catalog model plus a tool for custom model identifiers.

The library uses an existing browser session; it does not automate a browser. Session tokens are account credentials: never log or fixture real tokens, and do not expose the HTTP server to a network without understanding the risk. The API binds to localhost by default; --host 0.0.0.0 is an explicit operator choice.

Code map and request flow

• src/perplexity_webui_scraper/core/: public client (client.py), conversation state/lifecycle (conversation.py), uploads (files.py), payload construction (payload.py), SSE parsing (parser.py), response models (response.py), and account normalization/access (account.py).
• http/: curl-cffi transport, session cookie, fingerprinting, rate limiting, retries, and HTTP error translation. Keep transport/authentication mechanics here rather than reimplementing them in consumers.
• models/ + _static/models.json: Model, validation, and ordered registry. The JSON is the source of truth for catalog entries, public IDs, MCP tool names, tiers, modes, observed status, and generated model documentation.
• config/ and _internal/: typed configuration, shared aliases/internal values, endpoint constants, and exceptions.
• api/: per-request authentication, client/conversation caches, schemas, request conversion helpers, routes, and launcher. cli/ and mcp/ are adapters to the core; avoid duplicating business rules there.
• scripts/: CI path classification and generation/validation of derived documentation.
• tests/: offline tests with synthetic fixtures and mocks. Tests requiring network access or an account must be opt-in and must not be required by the normal suite.
• docs/, README.md, CHANGELOG.md: user documentation and release history. MkDocs builds the Read the Docs site. Catalog tables in docs/api-reference.md and docs/mcp-server.md are generated.
• Repository root: pyproject.toml holds Python configuration/dependencies; uv.lock is the lockfile; Justfile defines standard commands; Containerfile* build API/MCP images; .github/workflows/ validate, secure, and publish releases.