Borrowing it
Nothing to install: this file belongs to sharozdawa/robotstxt-ai. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/sharozdawa/robotstxt-ai/main/.claude/skills/robots-ai/SKILL.mdgit clone --depth 1 https://github.com/sharozdawa/robotstxt-aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai)<a href="https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai"><img src="https://agentmods.dev/badge/skills/sharozdawa/robotstxt-ai/robots-ai/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sharozdawa/robotstxt-ai/robots-ai"><img src="https://agentmods.dev/badge/skills/sharozdawa/robotstxt-ai/robots-ai.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00044 | $0.00837 |
| Opus 5 | $0.00022 | $0.00418 |
| Sonnet 5 | $0.00009 | $0.00167 |
| Haiku 4.5 | $0.00004 | $0.00084 |
Grade A, and why
robots-ai scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 88 lines — stays where its author put it; the contents beside it link to each section on GitHub.
robots-ai
Analyze, audit, and generate robots.txt files with full awareness of 20+ AI crawlers.
Capabilities
- Analyze any website's robots.txt to see which AI bots are blocked/allowed
- Generate a robots.txt with toggleable AI bot blocking
- Audit existing robots.txt for completeness and issues
- List all known AI crawlers with their user-agents, companies, and documentation links
AI Bots Database
You know about these AI crawlers and their user-agents:
| Bot | User-Agent | Company | Type |
|---|---|---|---|
| GPTBot | GPTBot | OpenAI | AI Crawler |
| ChatGPT-User | ChatGPT-User | OpenAI | AI Search |
| OAI-SearchBot | OAI-SearchBot | OpenAI | AI Search |
| ClaudeBot | ClaudeBot | Anthropic | AI Crawler |
| anthropic-ai | anthropic-ai | Anthropic | AI Crawler |
| Google-Extended | Google-Extended | AI Crawler | |
| PerplexityBot | PerplexityBot | Perplexity | AI Search |
| CCBot | CCBot | Common Crawl | AI Crawler |
| Bytespider | Bytespider | ByteDance | AI Crawler |
| Diffbot | Diffbot | Diffbot | AI Crawler |
| cohere-ai | cohere-ai | Cohere | AI Crawler |
| Amazonbot | Amazonbot | Amazon | AI Crawler |
| Meta-ExternalAgent | Meta-ExternalAgent | Meta | AI Crawler |
| Meta-ExternalFetcher | Meta-ExternalFetcher | Meta | AI Crawler |
| Applebot-Extended | Applebot-Extended | Apple | AI Crawler |
| YouBot | YouBot | You.com | AI Search |
| Timpibot | Timpibot | Timpi | AI Crawler |
| img2dataset | img2dataset | Open Source | AI Crawler |
Important Notes
- Google-Extended controls Gemini training access but does NOT affect Google Search indexing
- Blocking Googlebot removes the site from Google Search entirely — never do this unless explicitly asked
- CCBot feeds Common Crawl, which is used by many AI companies for training data
- Bytespider (ByteDance) and Timpibot are commonly blocked by default due to aggressive crawling
How to Analyze
When asked to analyze a robots.txt:
- Fetch the robots.txt from the URL (append /robots.txt if not included)
- Parse all User-agent directives and their Allow/Disallow rules
- Check each AI bot against the rules
- Report: which bots are blocked, which are allowed, and any issues found
- Suggest improvements if relevant
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 88 lines · 44 tokens per session scan A 36319e039069
robots-ai is a skill published in the GitHub repository sharozdawa/robotstxt-ai (1 stars, last pushed 5mo ago), licensed MIT. It adds 44 tokens to every session and 837 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
reskin
Author a NEW skin for the reskinnable-demo app. A skin is a self-contained domain plugin under src/skins/ / that implements the frozen Skin contract (src/shell/skin-contract.ts) to swap the app's entire experience — brand, theme, layout, pages, tools, data, and agent — as a live sales demo. Use when the user says "add…
copilotkit-setup
Use when adding CopilotKit to an existing project or bootstrapping a new CopilotKit project from scratch. Covers framework detection, package installation, runtime wiring (managed Intelligence or self-hosted SSE), provider setup, and first working chat integration.
setup-slack-channel
Use for the PROVIDER half of getting a locally running CopilotKit Channels agent to answer in Slack, when no Slack app exists yet — setting up a Channels bot in Slack for the first time, creating the Slack app and its tokens, attaching it to a managed Intelligence Channel, or when a Channel reports setuprequired, sits…
copilotkit-channels
Use for the CODE half of a managed Intelligence Channel with Slack or Microsoft Teams: customising the Channel a CLI-scaffolded project already ships, or — for a project the CLI did not generate — writing the Channel declaration, the long-running host, and the awaited activation call. Teams provider setup is in scope…
runtime
@copilotkit/runtime — mount a fetch-native CopilotRuntime on any JS server, wire middleware, pick an AgentRunner, instantiate BuiltInAgent (Factory Mode with TanStack AI is the preferred default) or plug in any of 12 external agent frameworks (Mastra, LangGraph, CrewAI Crews/Flows, PydanticAI, ADK, LlamaIndex, Agno…
copilotkit-integrations
Use when wiring an external agent framework (LangGraph, CrewAI, PydanticAI, Mastra, ADK, LlamaIndex, Agno, Strands, Microsoft Agent Framework, or others) into a CopilotKit application via the AG-UI protocol.