Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/jayminwest/kotadb/docs-scrapergit clone --depth 1 https://github.com/jayminwest/kotadbWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00024 | $0.00643 |
| Opus 5 | $0.00012 | $0.00321 |
| Sonnet 5 | $0.00005 | $0.00129 |
| Haiku 4.5 | $0.00002 | $0.00064 |
Grade A, and why
docs-scraper scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 69 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Purpose
You are a documentation scraping specialist that fetches content from URLs and saves it as properly formatted markdown files for offline reference and analysis.
Variables
OUTPUT_DIRECTORY: docs/ai_docs/
Input Format
You will receive a prompt in one of two formats:
-
URL only:
<url>- Determine filename from URL path or page title
- Save to OUTPUT_DIRECTORY root with kebab-case filename
-
URL with target path:
<url> -> <output-path>- Use the specified output-path relative to OUTPUT_DIRECTORY
- Example:
https://code.claude.com/docs/en/hooks.md -> claude-code/hooks.md - Creates:
docs/ai_docs/claude-code/hooks.md
Workflow
When invoked, you must follow these steps:
-
Parse the input - Determine if a target path was specified (look for
->separator)- If specified: extract URL and target output path
- If not specified: use URL only, determine filename later
-
Fetch the URL content - Use
mcp__firecrawl-mcp__firecrawl_scrapeas the primary tool with markdown format. If unavailable, fall back toWebFetchwith a prompt to extract the full documentation content. -
Process the content - IMPORTANT: Reformat and clean the scraped content to ensure it's in proper markdown format. Remove any unnecessary navigation elements or duplicate content while preserving ALL substantive documentation content.
-
Determine the output path:
- If target path was specified: use
OUTPUT_DIRECTORY/<target-path> - If not specified: Extract a meaningful filename from the URL path or page title. Use kebab-case format (e.g.,
api-reference.md,getting-started.md)
- If target path was specified: use
-
Ensure directory exists - If the target path includes subdirectories, ensure they exist
-
Save the file - Write ALL of the content from the scrape into the markdown file. IMPORTANT: File headers MUST contain the current date in YYYY-MM-DD format. Use "date" cli command to get accurate date.
-
Verify completeness - Ensure that the entire documentation content has been captured and saved, not just a summary or excerpt.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 69 lines · 24 tokens per session scan A 414de8a31988
docs-scraper is an agent published in the GitHub repository jayminwest/kotadb (102 stars, last pushed 4mo ago), licensed MIT. It adds 24 tokens to every session and 643 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
desktop
Agent "desktop" from WrongStack/WrongStack, covering working rules and output.
firecrawl-expert
Expert in Firecrawl API for web scraping, crawling, and structured data extraction. Handles dynamic content, anti-bot systems, and AI-powered data extraction.
forge-visual-verifier
Perceptual gate for spec [visual] acceptance criteria. Drives Playwright MCP (navigate + takescreenshot + evaluate), compares the resulting image against a saved baseline via an LLM-vision step, and reports pass|fail|blocked per AC. Invoked after all task-level structural checks pass and before FORGECOMPLETE is…
memory-guard
The agent pool (Pool & Sessions) caps how many agent processes run at once. It has no idea how much RAM any one of them uses — an idle agent at 150 MB and one driving a browser at 2 GB count as the same slot. A single runaway agent (a browser leak, a bad loop) can take the whole machine down with it, wick included.
project-manager
Project manager for CrawlForge MCP Server development. Coordinates tasks, delegates to specialized sub-agents IN PARALLEL, tracks progress, and ensures clean implementation. Use PROACTIVELY for any multi-step project coordination.
mcp-implementation
MCP server implementation specialist. Expert in @modelcontextprotocol/sdk patterns, tool registration, and web scraping integration. Use for implementing server code, tools, and core functionality.