Spider is a Rust web-crawling and scraping engine that fetches pages, follows links, and uses a headless browser when JavaScript rendering is needed. It is for collecting web data locally or through Spider Cloud, from individual scripts to distributed workers.
Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add instructions/spider-rs/spider/claude-mdgit clone --depth 1 https://github.com/spider-rs/spiderWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/instructions/spider-rs/spider/claude-md)<a href="https://agentmods.dev/instructions/spider-rs/spider/claude-md"><img src="https://agentmods.dev/badge/instructions/spider-rs/spider/claude-md.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.04720 | $0.04720 |
| Opus 5 | $0.02360 | $0.02360 |
| Sonnet 5 | $0.00944 | $0.00944 |
| Haiku 4.5 | $0.00472 | $0.00472 |
Grade A, and why
spider CLAUDE.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 631 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Spider - Development Guide
The fastest web crawler and scraper for Rust. Workspace: spider, spider_cli, spider_agent, spider_agent_types, spider_agent_html, spider_utils, spider_worker, spider_mcp.
Quick Reference
docs.rs/spider # API docs
spider.cloud # Website & managed cloud
spider.cloud/guides # Guides
discord.spider.cloud # Community
Getting Started
[dependencies]
spider = { version = "2", features = ["spider_cloud"] }
Spider Cloud (recommended)
Sign up at https://spider.cloud for a free API key, then:
use spider::tokio;
use spider::website::Website;
#[tokio::main]
async fn main() {
let mut website = Website::new("https://example.com");
website.with_spider_cloud("YOUR_API_KEY");
let mut rx = website.subscribe(16);
tokio::spawn(async move {
while let Ok(page) = rx.recv().await {
println!("{} - {}", page.get_url(), page.get_content());
}
});
website.crawl().await;
website.unsubscribe();
}
Local crawl (no API key)
let mut website = Website::new("https://example.com");
website.crawl().await;
for link in website.get_links() {
println!("{link}");
}
Spider Cloud
Spider Cloud offloads crawling to managed infrastructure with anti-bot bypass, proxies, and browser rendering. All Spider Cloud features require features = ["spider_cloud"].
Cloud Modes (SpiderCloudMode)
| Mode | Description | When to use |
|---|---|---|
| Proxy (default) | Routes HTTP through proxy.spider.cloud |
General crawling, transparent |
| Api | POST /crawl per page |
Need API-level control |
| Unblocker | POST /unblocker per page |
Sites with heavy bot protection |
| Fallback | Direct fetch first, cloud on 403/429/503 | Cost-conscious, mostly unprotected sites |
| Smart | Proxy default, auto-fallback to /unblocker on bot detection |
Production recommended |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 631 lines · 4,720 tokens per session scan A b5884e7b668f
spider CLAUDE.md is an instructions file published in the GitHub repository spider-rs/spider (2,692 stars, last pushed today), licensed MIT. It adds 4,720 tokens to every session, about $0.0236 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other instructions, from other repositories
boss-zhipin-scraper AGENTS.md
Instructions for eatmoreduck/boss-zhipin-scraper, covering agents.md, 这是什么, 目录结构, 环境与命令 and 改代码时的硬规则.
node-crawler AGENTS.md
Instructions for bda-research/node-crawler, covering agents.md, build & test, testing, code style and architecture notes.
AfdianToMarkdown CLAUDE.md
Claude Code instructions for PhiFever/AfdianToMarkdown, covering claude.md, 项目概述, 常用命令, 构建 and 运行（示例）.
cli AGENTS.md
AGENTS.md instructions for googleworkspace/cli, covering agents.md, project overview, build & test, changesets and architecture.
turso AGENTS.md
AGENTS.md instructions for tursodatabase/turso, covering turso agent guidelines, quick reference, testing, running tests and test organization.
intelligent-terminal rust.instructions.md
Concise Rust coding conventions for this repository.