manchittlab/TheCrawler

Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.

Open on GitHub

2Stars on the repository

1Mods indexed here, across every type

2mo agoLast push, which is what freshness is scored on

AGPL-3.0Licence, which decides whether bodies are shown

Sort Relevance Recently updated Fewest tokens Security grade Repo stars Stars this week Top rated 1 mods

thecrawler

manchittlab/TheCrawler

MCP server Claude CodeCodexCursor +2

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support. Runs locally from the thecrawler npm package.

2 2mo ago A tokens not measured AGPL-3.0