nolindnaidoo/scrape-le

Check whether a page is scrapeable before the scraper is written, and say when it cannot tell

This repository also configures its own agents. See what scrape-le tells them →

1Stars on the repository

8Mods indexed here, across every type

2d agoLast push, which is what freshness is scored on

MITLicence, which decides whether bodies are shown

MCP server Claude CodeCodexCursor +2

Analyse robots.txt content and report whether a path may be crawled. Runs locally from the scrape-le-mcp npm package.

not rated 1 2d ago A tokens not measured original MIT