robots-txt MCP servers

6 tagged robots-txt, measured the same way as everything else here.

next-devtools

01

thedaviddias/llms-txt-hub

MCP server Claude CodeCodexCursor +2

Next.js development tools MCP server with stdio transport. Runs locally from the next-devtools-mcp npm package.

not rated 916 +3 yesterday A tokens not measured

asish-singh/agent-readiness-auditor

MCP server Claude CodeCodexCursor +2

Audits any website for AI agent readiness and safety, scoring prompt injection risk and llms.txt. Runs locally from the agent-readiness-auditor npm package.

not rated 1 11d ago A tokens not measured original MIT

mcp-seo-audit

03

atomno-mcp/mcp-seo-audit

MCP server Claude CodeCodexCursor +2

MCP server for technical SEO audits powered by the detail.web engine: site health score, issues across 8 categories and a GEO (AI-search visibility) sub-score. Runs locally from the atomno-mcp-seo-audit Python package. Needs 1 environment variable to run.

not rated 1 1mo ago A tokens not measured original MIT

magpie-scraper

04

MBemera/magpie

MCP server Claude CodeCodexCursor +2

MCP server "magpie-scraper" as configured in MBemera/magpie. Runs locally from the magpie-scraper npm package.

not rated 1 3mo ago A tokens not measured original MIT

geo-inspector-mcp

05

Bigsupe55/geo-inspector-mcp

MCP server Claude CodeCodexCursor +2

Inspect a site's AI-search readiness: AI crawler access, llms.txt, schema markup, meta directives. Runs locally from the geo-inspector-mcp npm package.

not rated 0 1mo ago A tokens not measured original MIT

crawl-census

06

taylorsmithgg/crawl-census-client

MCP server Claude CodeCodexCursor +2

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it? Remote server at crawlcensus.com.

not rated 0 2mo ago A tokens not measured