agentmods.dev

seo-crawlability

seo-crawlability is a skill for Claude Code from Hainrixz/claude-seo-ai. It costs 72 tokens per session (1,818 once invoked), scanned A, original, MIT.

A website audit for checking whether search engines can reach and read a page and its supporting files. It examines robots.txt, which tells crawlers what they may access.

In plain words
What is it for?
Use it to check robots.txt availability and syntax, find blocked assets or content, review crawl-delay settings, confirm a sitemap declaration, and check access for Google and Bing crawlers.
Why use it?
It helps find access rules that accidentally block important content, styles, scripts, or fonts, preventing search engines from indexing the page correctly.

Skill for Claude Code

Written for Claude Code: allowed-tools in frontmatter.

Runs only inside its plugin — its command needs a path that Claude Code sets for a plugin’s own hooks and for nothing else. Install the plugin, not this.

Part of the claude-seo-ai plugin — 34 skills, 5 agents, 1 hook shipped together

not rated 59repo +11 changed 3d ago A scan 72 tokens original MIT

Good fit Use it to check robots.txt availability and syntax, find blocked assets or content, review crawl-delay settings, confirm a sitemap declaration, and check access for Google and Bing crawlers.

Compare 6 skills from other repositories ↓
View source ↗ Hainrixz/claude-seo-ai tododeia.com
Install

Getting it into your agent

This one installs as part of its plugin. Adding the marketplace and installing the plugin brings it with everything else the plugin ships.

Claude Code
/plugin marketplace add Hainrixz/claude-seo-ai
Claude Code
/plugin install claude-seo-ai

Made for: Claude Code.

Or install claude-seo-ai, the plugin that ships this one along with the rest of its 34 skills, 5 agents, 1 hook.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for seo-crawlability

README.md
[![agentmods](https://agentmods.dev/badge/skills/hainrixz/claude-seo-ai/seo-crawlability/github.svg)](https://agentmods.dev/skills/hainrixz/claude-seo-ai/seo-crawlability)
Your own site
<a href="https://agentmods.dev/skills/hainrixz/claude-seo-ai/seo-crawlability"><img src="https://agentmods.dev/badge/skills/hainrixz/claude-seo-ai/seo-crawlability/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for seo-crawlability

Your own site · 80×15
<a href="https://agentmods.dev/skills/hainrixz/claude-seo-ai/seo-crawlability"><img src="https://agentmods.dev/badge/skills/hainrixz/claude-seo-ai/seo-crawlability.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 72 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,818 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00072 $0.01818
Opus 5 $0.00036 $0.00909
Sonnet 5 $0.00014 $0.00364
Haiku 4.5 $0.00007 $0.00182

Measured 3d ago against content hash b2b759d39b91, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-11, from the pricing page.

Security

Grade A, and why

seo-crawlability scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/seo-crawlability/SKILL.md · 53 lines

How it starts

The opening of the file, as written. The whole thing — 53 lines — stays where its author put it; the contents beside it link to each section on GitHub.

seo-crawlability (M1)

Crawl access is the precondition for every other search signal: if Googlebot/Bingbot can't fetch the page and its assets, nothing else ranks. This module covers general-purpose crawl access only. AI-specific bot directives (GPTBot, Claude-SearchBot, etc.) live in seo-ai-crawlers (M14) and AI discovery files (llms.txt, agents.md, UCP) in seo-ai-discovery (M21); see references/ai-crawlers.md for that boundary.

Inputs

Work from the PageSnapshot named in your dispatch envelope: read parsed from <run_dir>/pages/<slug>.json plus status_chain and headers, and the parsed robots file in <run_dir>/site/robots.json (syntax, groups, Googlebot/Bingbot verdicts for the audited URL, Sitemap: lines); Grep pages/<slug>.html for verbatim evidence. Deterministic findings already emitted by audit.mjs are listed in <run_dir>/findings.deterministic.json — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs" and use the printed snapshot path.

Audits

Working from the PageSnapshot (parsed_rendered when render.used is not none, else parsed) plus site/robots.json:

  1. Reachability: /robots.txt returns 200 (a 404 means "allow all" but is worth flagging; a 5xx can suspend crawling).
  2. Syntax: each line is a valid directive (User-agent, Disallow, Allow, Sitemap, Crawl-delay); flag unknown tokens, missing User-agent group headers, and BOM/encoding issues.
  3. Asset blocking: any Disallow that blocks CSS/JS, fonts, or /wp-includes/-style paths — this breaks rendering and is a leading cause of "page looks broken to Google" (cross-check with M-render).
  4. Content blocking: Disallow rules that hide important indexable paths from Googlebot/Bingbot.
  5. Crawl-delay sanity: a large Crawl-delay (or one applied to the global group) can starve crawl budget; note that Googlebot ignores Crawl-delay but Bingbot honors it.
  6. Sitemap directive: presence of at least one absolute Sitemap: URL.
  7. Overall access: resolve the effective ruleset for Googlebot and Bingbot against the audited URL — does it end up allowed?

Read the full file on GitHub · 53 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 3d ago Changed · +7 lines b2b759d39b91
  2. 5d ago Changed · +3 lines a01a1084e598
  3. 11d ago First seen · 43 lines · 72 tokens per session scan A 5be452d8db27

Subscribe to this mod's changes

seo-crawlability is a skill published in the GitHub repository Hainrixz/claude-seo-ai (59 stars, last pushed 4d ago), licensed MIT. It adds 72 tokens to every session and 1,818 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Claude CodeMarketing and SEOaeoai-searchanswer-engine-optimizationclaude-pluginclaude-skills
Related

Other skills, from other repositories

linkedin-post

Draft a daily LinkedIn post for S M Moshiur Rahman about business development, grounded in a real problem he hit that day and real numbers from OmniRank. Use when he says "linkedin post", "today's post", "draft a post", or describes friction he wants written up.

bemoshiur/OmniRank · 63 tokens

audit

Use when asked to audit a site's SEO, check AEO or answer-engine readiness, diagnose why a page is not ranking or not being cited by AI, verify structured data, or run pre-deploy discoverability checks on built HTML.

bemoshiur/OmniRank · 49 tokens

geo-artifacts

Use when asked to generate or fix llms.txt, llms-full.txt or facts.json, make a site citable or ingestible by ChatGPT, Claude, Perplexity or Gemini, or publish machine-readable ground truth for AI crawlers.

bemoshiur/OmniRank · 54 tokens

web-optimization

Audit or generate web content optimized for traditional search (SEO), AI generative answer engines like ChatGPT/Perplexity/Google AI Overviews (GEO), and answer engines / featured snippets / voice (AEO). Use when the user asks to improve a page's ranking or AI-citability, run an SEO/GEO/AEO audit of a URL or file, add…

staksoft/geo-seo-aeo-skill · 105 tokens

fire-your-seo-agency

A procedure for improving how a website appears in search engines and how AI answer systems find and cite it. It covers search, answer-engine, generative-AI, and Naver visibility.

leopard627/fire-your-seo-agency · 162 tokens

ansvisor-aeo-coach-standalone

Standalone (no-MCP) version of the Ansvisor AEO Coach. Use this only when the user's Claude client cannot connect to the Ansvisor MCP server (e.g. claude.ai web without a Connector configured). Fetches live data from the Ansvisor REST API directly with the user's API key via code execution. For clients that support…

ansvisor/ansvisor · 124 tokens