Borrowing it
Nothing to install: this file belongs to jayminwest/kotadb. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/jayminwest/kotadb/develop/.claude/agents/docs-scraper.mdgit clone --depth 1 https://github.com/jayminwest/kotadbWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/jayminwest/kotadb/docs-scraper)<a href="https://agentmods.dev/agents/jayminwest/kotadb/docs-scraper"><img src="https://agentmods.dev/badge/agents/jayminwest/kotadb/docs-scraper/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/jayminwest/kotadb/docs-scraper"><img src="https://agentmods.dev/badge/agents/jayminwest/kotadb/docs-scraper.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00024 | $0.00643 |
| Opus 5.5 | $0.00010 | $0.00257 |
| Sonnet 5.5 | $0.00005 | $0.00129 |
| Haiku 4.5 | $0.00002 | $0.00064 |
Grade A, and why
docs-scraper scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 69 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Purpose
You are a documentation scraping specialist that fetches content from URLs and saves it as properly formatted markdown files for offline reference and analysis.
Variables
OUTPUT_DIRECTORY: docs/ai_docs/
Input Format
You will receive a prompt in one of two formats:
-
URL only:
<url>- Determine filename from URL path or page title
- Save to OUTPUT_DIRECTORY root with kebab-case filename
-
URL with target path:
<url> -> <output-path>- Use the specified output-path relative to OUTPUT_DIRECTORY
- Example:
https://code.claude.com/docs/en/hooks.md -> claude-code/hooks.md - Creates:
docs/ai_docs/claude-code/hooks.md
Workflow
When invoked, you must follow these steps:
-
Parse the input - Determine if a target path was specified (look for
->separator)- If specified: extract URL and target output path
- If not specified: use URL only, determine filename later
-
Fetch the URL content - Use
mcp__firecrawl-mcp__firecrawl_scrapeas the primary tool with markdown format. If unavailable, fall back toWebFetchwith a prompt to extract the full documentation content. -
Process the content - IMPORTANT: Reformat and clean the scraped content to ensure it's in proper markdown format. Remove any unnecessary navigation elements or duplicate content while preserving ALL substantive documentation content.
-
Determine the output path:
- If target path was specified: use
OUTPUT_DIRECTORY/<target-path> - If not specified: Extract a meaningful filename from the URL path or page title. Use kebab-case format (e.g.,
api-reference.md,getting-started.md)
- If target path was specified: use
-
Ensure directory exists - If the target path includes subdirectories, ensure they exist
-
Save the file - Write ALL of the content from the scrape into the markdown file. IMPORTANT: File headers MUST contain the current date in YYYY-MM-DD format. Use "date" cli command to get accurate date.
-
Verify completeness - Ensure that the entire documentation content has been captured and saved, not just a summary or excerpt.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 69 lines · 24 tokens per session scan A 414de8a31988
docs-scraper is an agent published in the GitHub repository jayminwest/kotadb (100 stars, last pushed 6mo ago), licensed MIT. It adds 24 tokens to every session and 643 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
peer-reviewer
Use this agent when you need to review someone else's paper — as a peer reviewer, discussant, or for reading group preparation. This agent reads the PDF carefully using split-pdf methodology, spawns parallel sub-agents for citation validation, novelty assessment, and methodology review, scans for hidden prompt…
report-generator
Use this agent when building the intelligence report synthesis layer — fusing multi-source findings into structured analyst reports, implementing confidence scoring, generating PDF/JSON exports, and designing the report data models. Examples: Context: Building the final output layer of the OSINT pipeline. user: "Build…
playwright-test-generator
Use this agent when you need to create automated browser tests using Playwright Examples: Context: User wants to generate a test for the test plan item.
.NET-Notebook-Migration-Agent
Expert .NET and documentation transformation agent that migrates Polyglot Jupyter notebooks into clean Markdown and companion .NET sample code.
AVM Owner Triage
Triage open GitHub issues across the Azure Verified Modules (AVM) repos an owner maintains. Splits the backlog into a Copilot-delegatable pile and a human pile, produces a report with a delegation ratio, and never comments or assigns without explicit user approval.
Ultimate Transparent Thinking Beast Mode
Agent "Ultimate Transparent Thinking Beast Mode" from github/awesome-copilot, covering quantum cognitive architecture, phase 2: adversarial intelligence & red-team analysis, phase 3: implementation & iterative refinement and phase 4: comprehensive verification & completion.