crawly-mcp: Instructions file for Codex

AGENTS.md

Project

crawly exposes two browser-backed operations — search and fetch — as MCP tools and a CLI for local LLM workflows. The implementation is built on patchright (a Playwright fork with bundled fingerprint patches), is async, and supports either a host Chromium binary or a patchright-managed Chromium bundle depending on runtime configuration.

The authoritative design document is docs/IMPLEMENTATION_PLAN.md. The user-facing overview is in README.md.

Naming

• Product name: crawly
• Python distribution: crawly-mcp
• Import package: crawly_mcp
• CLI executable: crawly-cli
• MCP server executable: crawly-mcp

Tooling

• uv — project and dependency manager. Use uv sync, uv add, uv add --dev, uv run <cmd>. Do not invoke pip directly.
• patchright + Chromium — browser automation via the patchright Playwright fork, which adds the fingerprint patches we depend on for search-provider traffic. Host mode uses a system Chromium binary; bundled mode uses patchright-managed Chromium (patchright install chromium).
• MCP SDK (mcp>=1.27.0) — tool server, stdio, SSE, and streamable-http transports.
• pydantic — request/response models and validation.
• beautifulsoup4 — HTML parsing for search result extraction.
• pytest + pytest-asyncio — test runner; async tests are standard.
• ruff — formatter and linter (black-compatible, line-length = 88). Configured in pyproject.toml with E W F I B UP SIM C90 N S A C4 PIE RET ARG PTH ERA PL TRY RUF plus isort for crawly_mcp as first-party.
• ast-index — native Rust CLI for structural code lookup across the Python tree. Use it as the primary code-search tool in this repo: prefer ast-index search, ast-index class, ast-index usages, and ast-index refs over ad-hoc grep/rg when locating symbols, call sites, or implementations. Keep the index fresh with ast-index update, or rebuild from scratch with ast-index rebuild after large refactors. Useful entry points: ast-index map, ast-index conventions, ast-index outline <file>, and ast-index changed --base main before review. Fall back to rg only for regex patterns, comment content, or string literals that are not indexed.
• loguru — runtime logging. All modules log through loguru.logger; the crawly_mcp._logging.configure_logging helper installs a single stderr sink (critical for stdio MCP transport) and intercepts stdlib logging so uvicorn and MCP SDK logs share the same sink. Level is controlled by CRAWLY_LOG_LEVEL (default INFO; accepted values TRACE DEBUG INFO WARNING ERROR CRITICAL). Log entry/exit of tool calls at INFO; reserve DEBUG for fine-grained parser/challenge/timing traces.