spider: Instructions file for Claude Code

Spider - Development Guide

The fastest web crawler and scraper for Rust. Workspace: spider, spider_cli, spider_agent, spider_agent_types, spider_agent_html, spider_utils, spider_worker, spider_mcp.

Quick Reference

docs.rs/spider          # API docs
spider.cloud            # Website & managed cloud
spider.cloud/guides     # Guides
discord.spider.cloud    # Community

Getting Started

[dependencies]
spider = { version = "2", features = ["spider_cloud"] }

Spider Cloud (recommended)

Sign up at https://spider.cloud for a free API key, then:

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    website.with_spider_cloud("YOUR_API_KEY");

    let mut rx = website.subscribe(16);
    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{} - {}", page.get_url(), page.get_content());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}

Local crawl (no API key)

let mut website = Website::new("https://example.com");
website.crawl().await;
for link in website.get_links() {
    println!("{link}");
}

---

Spider Cloud

Spider Cloud offloads crawling to managed infrastructure with anti-bot bypass, proxies, and browser rendering. All Spider Cloud features require features = ["spider_cloud"].

Cloud Modes (SpiderCloudMode)

Mode	Description	When to use
Proxy (default)	Routes HTTP through proxy.spider.cloud	General crawling, transparent
Api	POST /crawl per page	Need API-level control
Fallback	Direct fetch first, cloud on 403/429/503	Cost-conscious, mostly unprotected sites
Smart	Proxy default, auto-fallback to /scrape with stealth: true on bot detection	Production recommended

Configuration

use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};

// Full control via SpiderCloudConfig
let config = SpiderCloudConfig::new("sk-...")
    .with_mode(SpiderCloudMode::Smart)       // proxy + stealth scrape fallback
    .with_return_format("raw");              // "raw" = original HTML

let mut website = Website::new("https://example.com")
    .with_spider_cloud_config(config)
    .build()
    .unwrap();

// Or shorthand — defaults to Proxy mode
let mut website = Website::new("https://example.com");
website.with_spider_cloud("sk-...");