pixlie/SmartCrawler

A smart web crawler built in Rust that uses Claude AI to select the most relevant URLs from website sitemaps based on crawling objectives.

20Stars on the repository
1Mods indexed here, across every type
1y agoLast push, which is what freshness is scored on
GPL-3.0Licence, which decides whether bodies are shown

pixlie/SmartCrawler

Instructions file

Claude Code instructions for pixlie/SmartCrawler, covering claude.md, development workflow, core commands, build and run and build the project.

20 1y ago A 760 tokens GPL-3.0