robots txt mcp servers

5 tagged robots txt, measured the same way as everything else here.

next-devtools

01

thedaviddias/llms-txt-hub

MCP server Claude CodeCodexCursor +2

Next.js development tools MCP server with stdio transport. Runs locally from the next-devtools-mcp npm package.

900 13d ago A tokens not measured

asish-singh/agent-readiness-auditor

MCP server Claude CodeCodexCursor

Audits any website for AI agent readiness and safety, scoring prompt injection risk and llms.txt. Runs locally from the agent-readiness-auditor npm package.

1 1mo ago A tokens not measured original MIT

mcp-seo-audit

03

atomno-mcp/mcp-seo-audit

MCP server Claude CodeCodexCursor

MCP server for technical SEO audits powered by the detail.web engine: site health score, issues across 8 categories and a GEO (AI-search visibility) sub-score. Runs locally from the atomno-mcp-seo-audit Python package. Needs 1 environment variable to run.

1 1mo ago A tokens not measured original MIT

magpie-scraper

04

MBemera/magpie

MCP server Claude CodeCodexCursor

MCP server "magpie-scraper" as configured in MBemera/magpie. Runs locally from the magpie-scraper npm package.

1 1mo ago A tokens not measured original MIT

crawl-census

05

taylorsmithgg/crawl-census-client

MCP server Claude CodeCodexCursor

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? Remote server at crawlcensus.com.

0 9d ago A tokens not measured