Borrowing it
Nothing to install: this file belongs to sharozdawa/robotstxt-ai. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/sharozdawa/robotstxt-ai/main/.claude/skills/robots-ai/SKILL.mdgit clone --depth 1 https://github.com/sharozdawa/robotstxt-aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai)<a href="https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai"><img src="https://agentmods.dev/badge/skills/sharozdawa/robotstxt-ai/robots-ai/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai"><img src="https://agentmods.dev/badge/skills/sharozdawa/robotstxt-ai/robots-ai.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00044 | $0.00837 |
| Opus 5.5 | $0.00018 | $0.00335 |
| Sonnet 5.5 | $0.00009 | $0.00167 |
| Haiku 4.5 | $0.00004 | $0.00084 |
Grade A, and why
robots-ai scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 88 lines — stays where its author put it; the contents beside it link to each section on GitHub.
robots-ai
Analyze, audit, and generate robots.txt files with full awareness of 20+ AI crawlers.
Capabilities
- Analyze any website's robots.txt to see which AI bots are blocked/allowed
- Generate a robots.txt with toggleable AI bot blocking
- Audit existing robots.txt for completeness and issues
- List all known AI crawlers with their user-agents, companies, and documentation links
AI Bots Database
You know about these AI crawlers and their user-agents:
| Bot | User-Agent | Company | Type |
|---|---|---|---|
| GPTBot | GPTBot | OpenAI | AI Crawler |
| ChatGPT-User | ChatGPT-User | OpenAI | AI Search |
| OAI-SearchBot | OAI-SearchBot | OpenAI | AI Search |
| ClaudeBot | ClaudeBot | Anthropic | AI Crawler |
| anthropic-ai | anthropic-ai | Anthropic | AI Crawler |
| Google-Extended | Google-Extended | AI Crawler | |
| PerplexityBot | PerplexityBot | Perplexity | AI Search |
| CCBot | CCBot | Common Crawl | AI Crawler |
| Bytespider | Bytespider | ByteDance | AI Crawler |
| Diffbot | Diffbot | Diffbot | AI Crawler |
| cohere-ai | cohere-ai | Cohere | AI Crawler |
| Amazonbot | Amazonbot | Amazon | AI Crawler |
| Meta-ExternalAgent | Meta-ExternalAgent | Meta | AI Crawler |
| Meta-ExternalFetcher | Meta-ExternalFetcher | Meta | AI Crawler |
| Applebot-Extended | Applebot-Extended | Apple | AI Crawler |
| YouBot | YouBot | You.com | AI Search |
| Timpibot | Timpibot | Timpi | AI Crawler |
| img2dataset | img2dataset | Open Source | AI Crawler |
Important Notes
- Google-Extended controls Gemini training access but does NOT affect Google Search indexing
- Blocking Googlebot removes the site from Google Search entirely — never do this unless explicitly asked
- CCBot feeds Common Crawl, which is used by many AI companies for training data
- Bytespider (ByteDance) and Timpibot are commonly blocked by default due to aggressive crawling
How to Analyze
When asked to analyze a robots.txt:
- Fetch the robots.txt from the URL (append /robots.txt if not included)
- Parse all User-agent directives and their Allow/Disallow rules
- Check each AI bot against the rules
- Report: which bots are blocked, which are allowed, and any issues found
- Suggest improvements if relevant
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2mo ago First seen · 88 lines · 44 tokens per session scan A 36319e039069
robots-ai is a skill published in the GitHub repository sharozdawa/robotstxt-ai (1 stars, last pushed 8mo ago), licensed MIT. It adds 44 tokens to every session and 837 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
robots-ai compared with skills from other repositories
| Mod | Type | Stars | Last push | Licence | Scan | Tokens | Works with |
|---|---|---|---|---|---|---|---|
| robots-aisharozdawa/robotstxt-ai | Skill | 1 | 8mo ago | MIT | A | 44 | Claude Code |
| reskinCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 154 | Claude Code |
| copilotkit-channelsCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 112 | Claude Code |
| setup-slack-channelCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 206 | Claude Code |
| a2ui-catalogCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 190 | Claude Code |
| copilotkit-cliCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 87 | Claude Code |
| a2ui-coreCopilotKit/CopilotKit | Skill | 39k | yesterday | MIT | A | 18 | Claude Code |