Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add instructions/teslashibe/web-scrape/agents-mdgit clone --depth 1 https://github.com/teslashibe/web-scrapeWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00592 | $0.00592 |
| Opus 5 | $0.00296 | $0.00296 |
| Sonnet 5 | $0.00118 | $0.00118 |
| Haiku 4.5 | $0.00059 | $0.00059 |
Grade A, and why
web-scrape AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 63 lines — stays where its author put it; the contents beside it link to each section on GitHub.
AGENTS.md — web-scrape
Guidance for autonomous agents using this package.
Prefer CLI
web-scrape fetch <url> --json
web-scrape scrape <url> --json
web-scrape serve --port 8091
- Use
fetchwhen you need rendered DOM text (JS sites, soft challenges). - Use
scrapewhen HTTP HTML extract is enough (faster, no Chromium). - Use
serveonly when an MCP client must discover tools over HTTP.
Exit codes: 0 ok, 2 structured failure (status ≠ ok), 1 hard error.
Library
import { scrape, browserFetchURL } from "web-scrape";
Do not import server.js unless you intend to bind a port.
Status contract
Never treat non-ok bodies as page content. Important statuses:
| status | meaning |
|---|---|
ok |
trustworthy extract |
cloudflare_blocked / turnstile_required |
CF/Turnstile gate (solve only if CapSolver env is set) |
captcha_required |
reCAPTCHA/hCaptcha (optional same-origin provider; off by default) |
robots_disallowed |
robots.txt denied |
blocked / login_required / rate_limited |
hard gates |
challenge_provider_unavailable |
CapSolver/provider failed |
timeout / error |
transport |
Warnings may include turnstile:<code> — codes are secret-free. Never echo env secrets.
CapSolver rules
- Opt-in:
WEB_FETCH_TURNSTILE_PROVIDER=capsolver+WEB_FETCH_TURNSTILE_API_KEY. - Sticky proxy required for Cloudflare clearance (
WEBSHARE_USERNAME_TEMPLATE={user}-{country}-1). - CapSolver proxy encoding is socks5; do not “fix” that to HTTP.
- Do not send challenge-page
htmlonAntiCloudflareTask— CapSolver returnsinvalid html. - Budget: hard CF clears often need 30–90s; set
BROWSER_FETCH_TIMEOUT_MS≥ 120000 when enabled.
Safety
- Only public
http(s)URLs (SSRF guard). - No social-login / cookie jar reuse — every browser context is ephemeral.
- Do not disable robots checks.
- Do not log API keys, proxy URLs, or solved tokens.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 63 lines · 592 tokens per session scan A 4b1556b007b7
web-scrape AGENTS.md is an instructions file published in the GitHub repository teslashibe/web-scrape (1 stars, last pushed 1mo ago), licensed Apache-2.0. It adds 592 tokens to every session, about $0.0030 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other instructions, from other repositories
Browser4 CLAUDE.md
Instructions for platonai/Browser4, covering browser4 — project context for claude, architecture, key dispatch chain (cli → browser), batch commands and e2e test structure.
spring-ai-agentcore AGENTS.md
Instructions for spring-ai-community/spring-ai-agentcore, covering agents.md, project overview, architecture, key components and artifact store classes.
flyto-core CLAUDE.md
Instructions for flytohub/flyto-core, covering claude notes, cross-agent handoff and shared code intelligence.
deckforge AGENTS.md
Instructions for tph-kds/deckforge, covering deckforge agent entry point, code intelligence, read order, default routing and non-negotiable implementation rules.
fast-browser CLAUDE.md
Claude Code instructions for m4ttstack/fast-browser, covering fast browser plugin, where a change belongs, fork branch: use fast-browser-runtime, releasing a new runtime and re-pinning this repo: use the script.
agentoscope AGENTS.md
Instructions for rafaelcg/agentoscope, covering agentoscope — agent instructions, what this is, layout, commands and conventions.