taylorsmithgg/crawl-census-client

Ask before you fetch. Skip domains that will refuse your crawler, and never route around an HTTP 402 paywall. JS + Python, zero dependencies.

0Stars on the repository

1Mods indexed here, across every type

2mo agoLast push, which is what freshness is scored on

customA LICENSE file GitHub cannot name, so bodies are not copied

MCP server Claude CodeCodexCursor +2

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? Remote server at crawlcensus.com.

not rated 0 2mo ago A tokens not measured