Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add JakeLabate/Claude-SEO-Skills --skill robots-txt-auditgit clone --depth 1 https://github.com/JakeLabate/Claude-SEO-SkillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/jakelabate/claude-seo-skills/robots-txt-audit)<a href="https://agentmods.dev/skills/jakelabate/claude-seo-skills/robots-txt-audit"><img src="https://agentmods.dev/badge/skills/jakelabate/claude-seo-skills/robots-txt-audit/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/jakelabate/claude-seo-skills/robots-txt-audit"><img src="https://agentmods.dev/badge/skills/jakelabate/claude-seo-skills/robots-txt-audit.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00160 | $0.01791 |
| Opus 5 | $0.00080 | $0.00896 |
| Sonnet 5 | $0.00032 | $0.00358 |
| Haiku 4.5 | $0.00016 | $0.00179 |
Grade A, and why
robots-txt-audit scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 129 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Robots.txt Audit
Audit a website's robots.txt file and produce an actionable SEO report.
When to use this skill
Use this skill when the user asks to:
- Audit, validate, or check a robots.txt file or crawl directives
- Find out why pages are not being crawled or why Google says a URL is "blocked by robots.txt"
- Check that important pages aren't blocked and that CSS/JS resources can be rendered
- Confirm robots.txt declares the XML sitemap and returns 200 as plain text
- Catch a stray
Disallow: /that took the whole site out of search - Clean up syntax errors and unsupported directives (
noindex,crawl-delay, etc.)
Inputs to collect
Before starting, confirm with the user:
- Site URL or robots.txt URL — a live site root (e.g.,
https://example.com, which fetches/robots.txt) or a specific robots.txt URL (--robots). - Resource check — whether to fetch the homepage and test its CSS/JS/image references against the rules (default on). Use
--no-resourcesto skip it.
What robots.txt does and does not do: robots.txt controls crawling, not indexing. A
Disallowstops compliant crawlers from fetching a page, but a blocked URL can still be indexed (without a snippet) if other pages link to it. To keep a page out of the index, allow crawling and use anoindexmeta tag orX-Robots-Tagheader instead —noindexinside robots.txt is unsupported and ignored. Most checks here protect crawl access and rendering rather than indexing.
Workflow
Step 1: Fetch and parse robots.txt
Use scripts/collect_robots.py:
python3 scripts/collect_robots.py https://example.com --output robots_inventory.json
Point directly at a robots.txt with --robots:
python3 scripts/collect_robots.py https://example.com/robots.txt --robots --output robots_inventory.json
The script fetches /robots.txt without auto-following redirects (so a redirected or HTML-served file is visible), records HTTP status, content type, byte size, gzip, and a BOM flag, then parses user-agent groups, Sitemap: directives, comments, unknown/unsupported directives, and per-line syntax errors. It probes every declared sitemap URL for status and (unless --no-resources) fetches the homepage to test same-host CSS/JS/image references against the * group's rules.
What ships with it
6 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 12d ago First seen · 129 lines · 0 tokens per session scan A 8c7ef725abaf
robots-txt-audit is a skill published in the GitHub repository JakeLabate/Claude-SEO-Skills (2 stars, last pushed 2mo ago), licensed MIT. It adds 160 tokens to every session and 1,791 once invoked, about $0.0008 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
seo-audit
A checklist-based SEO review for a website. SEO, or search engine optimization, is the work of improving a site so search engines can understand and rank it.
fire-your-seo-agency
A procedure for improving how a website appears in search engines and how AI answer systems find and cite it. It covers search, answer-engine, generative-AI, and Naver visibility.
geo-loop
Run one bounded eGEOagents loop iteration over a workspace domain - read the charter and fresh collector data, do ONE unit of work, write substrate artifacts, append one Timeline entry and one LOG line. Use for loop mode, /geo:loop, scheduled GEO runs, or continuous monitoring.
content-scoring
Score content against the 10 GEO criteria with evidence and prioritized fixes. Use when users ask to score, rate, evaluate, or estimate ranking strength.
competitive-analysis
Analyze AI-search competitors for a query and recommend ranking strategy. Use when users ask competitor analysis, who ranks, or competitive landscape.
validation-doctor
Check Brave Search and Chrome DevTools MCP availability and provide exact setup snippets. Use when validation dependencies are missing or uncertain.