Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add instructions/stretchcloud/deepscrape/agents-mdgit clone --depth 1 https://github.com/stretchcloud/deepscrapeWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00854 | $0.00854 |
| Opus 5 | $0.00427 | $0.00427 |
| Sonnet 5 | $0.00171 | $0.00171 |
| Haiku 4.5 | $0.00085 | $0.00085 |
Grade A, and why
deepscrape AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 61 lines — stays where its author put it; the contents beside it link to each section on GitHub.
DeepScrape Agent Notes
Overview
DeepScrape is a TypeScript/Node.js web scraping service built on Express. It provides API endpoints for single-page scraping, schema-based LLM extraction, batch scraping, URL discovery (/api/map), and multi-page crawling with a Redis-backed BullMQ queue. Playwright is the primary scraper with an HTTP fallback, and content can be returned as HTML, Markdown, or text.
Architecture Summary
Entry point: src/index.ts initializes Express, middleware, routes, and the crawl queue worker.
Scrape flow: /api/scrape in src/api/routes/scraper.ts calls ScraperManager in src/scraper/scraper-manager.ts.
Process: Playwright scrape → HTTP fallback if Playwright fails → HTML cleaning → optional Markdown/text transform → optional LLM extraction → cache write.
LLM extraction: src/transformers/llm-extractor.ts uses OpenAIService from src/services/openai.service.ts.
Config: OPENAI_API_KEY, OPENAI_MODEL (defaults to gpt-4o).
Crawl flow: /api/crawl in src/api/controllers/crawler.controller.ts uses WebCrawler from src/scraper/crawler.ts.
Queueing: jobs are managed via src/services/queue.service.ts using BullMQ + Redis.
Streaming discovery: CrawlKickoffService in src/services/crawl-kickoff.service.ts can run streaming URL discovery via URLDiscoveryService and enqueue results as they stream in.
URL discovery: /api/map in src/api/routes/map.routes.ts uses URLDiscoveryService for sitemaps, robots.txt, common paths, search, and optional browser crawling.
Batch scraping: /api/batch/scrape in src/api/routes/batch-scrape.routes.ts uses BatchScrapeService in src/services/batch-scrape.service.ts with Redis persistence and optional webhooks.
Key Data Stores
Redis: crawl metadata, job status, batch metadata, and job results.
File cache: CacheService stores scrape results on disk (CACHE_DIRECTORY).
Logs: written under logs/ via Winston and morgan.
Important Files
Server bootstrap: src/index.ts
Routes: src/api/routes/*.ts
Controllers: src/api/controllers/*.ts
Scraper: src/scraper/*
Transformers: src/transformers/*
Services: src/services/*
Types: src/types/*
Utilities: src/utils/*
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 61 lines · 854 tokens per session scan A 240845e5f2e5
deepscrape AGENTS.md is an instructions file published in the GitHub repository stretchcloud/deepscrape (309 stars, last pushed 11d ago), licensed MIT. It adds 854 tokens to every session, about $0.0043 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other instructions, from other repositories
Browser4 CLAUDE.md
Instructions for platonai/Browser4, covering browser4 — project context for claude, architecture, key dispatch chain (cli → browser), batch commands and e2e test structure.
spring-ai-agentcore AGENTS.md
Instructions for spring-ai-community/spring-ai-agentcore, covering agents.md, project overview, architecture, key components and artifact store classes.
flyto-core CLAUDE.md
Instructions for flytohub/flyto-core, covering claude notes, cross-agent handoff and shared code intelligence.
metamask-extension AGENTS.md
AGENTS.md instructions for MetaMask/metamask-extension, covering agents.md, agent instructions summary, critical rules for agents, comprehensive guidelines location and quick setup.
deckforge AGENTS.md
Instructions for tph-kds/deckforge, covering deckforge agent entry point, code intelligence, read order, default routing and non-negotiable implementation rules.
fast-browser CLAUDE.md
Claude Code instructions for m4ttstack/fast-browser, covering fast browser plugin, where a change belongs, fork branch: use fast-browser-runtime, releasing a new runtime and re-pinning this repo: use the script.