Get web data for AI agents and LLMs - fast, efficient, and reliable with Rust
Spider is a Rust web-crawling and scraping engine that fetches pages, follows links, and uses a headless browser when JavaScript rendering is needed. It is for collecting web data locally or through Spider Cloud, from individual scripts to distributed workers.
Latest release v2.52.2 — v2.52.2 — CLI authenticate for Spider Cloud · 31 Mar 2026
These files are spider-rs/spider's own configuration. They tell Claude Code how to work on this repository, so they are not mods to install elsewhere. Copy one as a starting point and replace the parts that are about this project.
CLAUDE.md A 4,720 tok