docs-crawler

docs-crawler is a skill for Claude Code from CaesiumY/ko-design-md. It costs 256 tokens per session (1,483 once invoked), scanned A, original, MIT.

A crawler that collects an entire documentation or design-system website into one Markdown collection for an AI model to read. It finds pages through a sitemap or same-site links, extracts the main text, and can download images.

In plain words
What is it for?
Use it to capture documentation, design-system references, and their images from a website when preparing context for an AI workflow.
Why use it?
Reading a multi-page reference site manually is slow and can miss important pages. The crawler gathers the material into one local corpus for research, archiving, or later processing.

Skill for Claude Code

Written for Claude Code: installed under .claude/. Also seen: reads .claude/ paths.

Good fit Use it to capture documentation, design-system references, and their images from a website when preparing context for an AI workflow.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/caesiumy/ko-design-md/docs-crawler
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add CaesiumY/ko-design-md --skill docs-crawler
Clone the repo
git clone --depth 1 https://github.com/CaesiumY/ko-design-md

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for docs-crawler

README.md
[![agentmods](https://agentmods.dev/badge/skills/caesiumy/ko-design-md/docs-crawler/github.svg)](https://agentmods.dev/skills/caesiumy/ko-design-md/docs-crawler)
Your own site
<a href="https://agentmods.dev/skills/caesiumy/ko-design-md/docs-crawler"><img src="https://agentmods.dev/badge/skills/caesiumy/ko-design-md/docs-crawler/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for docs-crawler

Your own site · 80×15
<a href="https://agentmods.dev/skills/caesiumy/ko-design-md/docs-crawler"><img src="https://agentmods.dev/badge/skills/caesiumy/ko-design-md/docs-crawler.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 256 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,483 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector pass 7 Sept 2026
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00256 $0.01483
Opus 5.5 $0.00102 $0.00593
Sonnet 5.5 $0.00051 $0.00297
Haiku 4.5 $0.00026 $0.00148

Measured 1mo ago against content hash 4fe3b326cae1, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

docs-crawler scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

The scan reads SKILL.md. This mod also ships 14 executable files (scripts/crawl.ts, scripts/crawl/args.test.ts, scripts/crawl/args.ts, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.claude/skills/docs-crawler/SKILL.md · 136 lines

How it starts

The opening of the file, as written. The whole thing — 136 lines — stays where its author put it; the contents beside it link to each section on GitHub.

docs-crawler

Crawl a documentation website into a single Markdown corpus that an LLM can consume as context.

When to use

The user wants to capture a multi-page documentation, design-system, or reference site as Markdown — to feed an LLM, archive it, or research a design system. This skill is also invoked by the design-md skill's research phase.

Inputs

  • Site URL (required) — the documentation site, e.g. https://socarframe.socar.kr/. Any page on the site works; page discovery starts from the site's sitemap.xml.
  • Output directory (optional) — where to write the corpus. Defaults to ./{host}-docs/ in the current working directory.

If the user has not provided a URL, ask for one — do not guess.

Running the crawl

From the repository root:

pnpm crawl:docs <site-url> [--out <dir>] [--external-images]

Equivalent direct form:

pnpm exec tsx .claude/skills/docs-crawler/scripts/crawl.ts <site-url> [--out <dir>]

The engine discovers pages via sitemap.xml (falling back to same-origin link-following), fetches each page, extracts the main content, and converts it to Markdown. It prints per-page progress and a final summary line.

By default it also localizes images: every external image and inline base64 data: image is downloaded into crawl/images/ and the Markdown is rewritten to relative paths, so the corpus is self-contained for renderers that can't fetch external URLs (Claude Design, offline previews). Pass --external-images to skip downloading and keep the original external URLs instead.

Output

These artifacts are written under the output directory:

  • crawl-corpus.md — every successful page merged into one document, with a table of contents and a Source: URL per page. This is the primary deliverable.
  • crawl/pages/{NNN}-{slug}.md — one file per page, each with source_url / title / method frontmatter.
  • crawl/images/ — downloaded image files (skipped with --external-images), referenced from the corpus and page files by relative paths.
  • crawl/manifest.json — a per-URL audit log (status, fetch method, extracted character count, error reason).

Read the full file on GitHub · 136 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 1mo ago First seen · 136 lines · 256 tokens per session scan A 4fe3b326cae1

Subscribe to this mod's changes

docs-crawler is a skill published in the GitHub repository CaesiumY/ko-design-md (65 stars, last pushed yesterday), licensed MIT. It adds 256 tokens to every session and 1,483 once invoked, about $0.0010 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

omd-apply

A skill that applies a project's DESIGN.md file as visual and brand guidance for interface work. It covers components, colors, fonts, layout, copy, motion, and visual assets.

kwakseongjae/oh-my-design · 129 tokens

bench-fixture-gen

Generates the shared per-task fixture that every competing arm starts from: a realistic dataset plus an image asset base. Fairness rule: fixtures are generated ONCE per task by this preprocessor and sealed (sha256) before any arm runs; arms never generate their own.

kwakseongjae/oh-my-design · 0 tokens

design-engineering

Premium design engineering skill for agentic workflows — produces high-end, distinctive UI designs using DESIGN.md as the portable contract across Pencil MCP (in-IDE canvas), Figma MCP (team handoff + design tokens), and Google Stitch (vibe exploration + AI generation). Enforces anti-generic principles, WCAG 2.2 AA…

broomva/skills · 231 tokens

design-md

Pick a design language from a categorized offline catalog of 152 entries (84 brand-inspired DESIGN.md files and 68 generic styles), write it as DESIGN.md in the project root, then implement or restyle production UI from that file and verify it in a running page. Use when the user asks to list or compare design styles…

bahayonghang/my-ai-cli-toolkit · 225 tokens

dembrandt

A TypeScript command-line tool that renders a web page with Playwright and examines its actual styles. It turns colours, typography, spacing, borders, shadows, animation curves, components, and breakpoints into structured design tokens.

AvalonLee/AstraCraft-Program · 0 tokens

designlang-tokens

Use when styling UI for cal.com — references the extracted design system tokens instead of inventing colors, spacing, or typography.

Manavarya09/design-extract · 30 tokens