A smart web crawler built in Rust that uses Claude AI to select the most relevant URLs from website sitemaps based on crawling objectives.
Latest release v0.4.1 — SmartCrawler 0.4.1 · 9 Jul 2025
1 file for Claude Code: SmartCrawler CLAUDE.md — 760 tokens loaded in every session.
CLAUDE.md A 760 tok These files are pixlie/SmartCrawler's own configuration — they tell Claude Code how to work on this repository, so they are not mods to install elsewhere. Copy one as a starting point and replace the parts that are about this project.