Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add AmbroseX/larksnap --skill docs-scrapergit clone --depth 1 https://github.com/AmbroseX/larksnapWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ambrosex/larksnap/docs-scraper)<a href="https://agentmods.dev/skills/ambrosex/larksnap/docs-scraper"><img src="https://agentmods.dev/badge/skills/ambrosex/larksnap/docs-scraper/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ambrosex/larksnap/docs-scraper"><img src="https://agentmods.dev/badge/skills/ambrosex/larksnap/docs-scraper.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00158 | $0.01293 |
| Opus 5.5 | $0.00063 | $0.00517 |
| Sonnet 5.5 | $0.00032 | $0.00259 |
| Haiku 4.5 | $0.00016 | $0.00129 |
Grade A, and why
docs-scraper scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
# 第 1/3 条：整包拆单页（先自己 curl 下来） How it starts
The opening of the file, as written. The whole thing — 86 lines — stays where its author put it; the contents beside it link to each section on GitHub.
docs-scraper
抓文档站的关键不是写爬虫，是先探路。现代文档站为了喂 AI，大多主动提供了原始 Markdown 入口，能直接拿到源码就别去解析 HTML。
第一步永远是探测，不要跳过。
python3 scripts/probe_docs.py https://目标站/某个文档页
它会认框架、把下面 7 条路挨个试一遍，告诉你哪条通。
选路阶梯（越靠前越省事）
| # | 捷径 | 一次拿到 | 用哪个脚本 |
|---|---|---|---|
| 1 | llms-full.txt |
整站全文，1 次请求 | split_bundle.py |
| 2 | 页面地址加 .md |
单页原始 Markdown | crawl_md.py |
| 3 | MkDocs search/search_index.json |
整站纯文本，1 次请求 | split_bundle.py |
| 4 | Sphinx _sources/*.rst.txt |
单页原始 rST | crawl_html.py（走 sitemap） |
| 5 | __NEXT_DATA__ 里的正文字段 |
单页 Markdown | crawl_next.py |
| 6 | sitemap.xml 出 URL 清单 |
只有清单 | crawl_html.py --sitemap |
| 7 | 顺着链接 BFS | 兜底 | crawl_html.py --seed |
第 3 条有坑：MkDocs 索引存的是剥掉标签的纯文本，代码块、链接、表格全丢。 实测 uv 84 个页面 0 个代码块，vLLM 同页面对比 25 个代码块 → 0 个。 只在明确"仅做全文检索"时用它，要存档一律走第 6 条。
四个抓取脚本
所有脚本都是并发 BFS + 落盘带 YAML 头的 Markdown，另外生成 _manifest.json 和 _index.md。
# 第 1/3 条：整包拆单页（先自己 curl 下来）
python3 scripts/split_bundle.py --in bundle.txt --out ./out
# 第 2 条：抓 .md 兄弟文件（Stripe/react.dev/vite/HuggingFace/GitBook/Mintlify）
python3 scripts/crawl_md.py --seed https://docs.stripe.com/payments \
--scope /payments --out ./out --concurrency 8
# 第 5 条：解析 __NEXT_DATA__（Palantir 这类多语言 Next.js 站）
python3 scripts/crawl_next.py --seed https://www.palantir.com/docs/zh/foundry/xxx/ \
--scope /zh/foundry/ --out ./out --resume
# 第 6/7 条：HTML 转 Markdown（兜底，覆盖其余全部）
~/miniconda3/bin/python3 scripts/crawl_html.py \
--sitemap https://docs.vllm.ai/sitemap.xml --scope /en/latest/ --out ./out
# 没有 sitemap 的站（比如 Tailwind）改用 --seed 顺着链接爬
~/miniconda3/bin/python3 scripts/crawl_html.py \
--seed https://tailwindcss.com/docs/flex --scope /docs --out ./out
公共参数：--concurrency（默认 8）、--limit（先小跑几页验证选路对不对）、
--delay（每线程每页间隔，默认 0.15 秒，别把人家服务器打疼）。
crawl_next.py 还有 --resume 断点续抓。
依赖
前五个脚本纯标准库，任何 python3 都能跑。
crawl_html.py 需要 bs4 + lxml。本机走 ~/miniconda3/bin/python3（已装）。
换环境可用 DOCS_SCRAPER_PY3 环境变量指定解释器。
动手前必读
What ships with it
8 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 86 lines · 158 tokens per session scan A beb839d256ca
docs-scraper is a skill published in the GitHub repository AmbroseX/larksnap (301 stars, last pushed 2d ago), licensed Apache-2.0. It adds 158 tokens to every session and 1,293 once invoked, about $0.0006 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-10-21.
Other skills, from other repositories
llmwiki
Build and use a persistent, citation-traceable knowledge wiki from documents, research papers, notes, or project documentation. Use for llmwiki projects, retrieving reusable evidence for an agent task, or compiling a corpus for repeated questions. A one-off file read or live web search does not need a wiki.
bailian-kb
A command-line manager for Alibaba Cloud Bailian knowledge bases, which are collections of documents prepared for search and question answering. It handles the stored documents, search services, text chunks, and data-centre files rather than everyday searches.
bili-note
A tool for turning Bilibili videos and posts into readable Markdown notes. Bilibili is a Chinese video and social-content platform; the tool can work with subtitles, article text, images, code, and selected comments.
rag-assistant
A local knowledge-base assistant for answering questions from indexed private documents. It searches the stored documents, checks whether the results are relevant, and cites the source files in its answers.
vibeknow-doc
Upload documents to vectoria and check processing status. Use when: user wants to upload a document, check if a document is ready, or get a docid for use with vibeknow create.
rag-prep
A document-preparation skill that cleans and structures source files for retrieval systems, which find relevant passages from a document collection.