What is web-scrape AGENTS.md?

A set of instructions for using the web-scrape package to fetch or extract text from websites, including sites rendered with JavaScript or protected by access checks.

Fetching rendered page text, scraping ordinary HTML, running the package as an HTTP service, or using its JavaScript library.

It helps an agent choose the appropriate command and distinguish trustworthy page content from blocks, login pages, robot restrictions, and other failures.

How teslashibe/web-scrape configures Codex and OpenCode

Reuse

Borrowing it

Nothing to install: this file belongs to teslashibe/web-scrape. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/teslashibe/web-scrape/main/AGENTS.md

Clone the repo

git clone --depth 1 https://github.com/teslashibe/web-scrape

Made for: Codex, OpenCode.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/teslashibe/web-scrape/agents-md/github.svg)](https://agentmods.dev/instructions/teslashibe/web-scrape/agents-md)

Your own site

<a href="https://agentmods.dev/instructions/teslashibe/web-scrape/agents-md"><img src="https://agentmods.dev/badge/instructions/teslashibe/web-scrape/agents-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/teslashibe/web-scrape/agents-md"><img src="https://agentmods.dev/badge/instructions/teslashibe/web-scrape/agents-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 592 This file is loaded in full into every session.

When invoked 592 The same file — it is already loaded in full.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00592	$0.00592
Opus 5.5	$0.00237	$0.00237
Sonnet 5.5	$0.00118	$0.00118
Haiku 4.5	$0.00059	$0.00059

Measured 2mo ago against content hash 4b1556b007b7, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

web-scrape AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

AGENTS.md · 63 lines

How it starts

The opening of the file, as written. The whole thing — 63 lines — stays where its author put it; the contents beside it link to each section on GitHub.

AGENTS.md — web-scrape

Guidance for autonomous agents using this package.

Prefer CLI

web-scrape fetch <url> --json
web-scrape scrape <url> --json
web-scrape serve --port 8091

Use fetch when you need rendered DOM text (JS sites, soft challenges).
Use scrape when HTTP HTML extract is enough (faster, no Chromium).
Use serve only when an MCP client must discover tools over HTTP.

Exit codes: 0 ok, 2 structured failure (status ≠ ok), 1 hard error.

Library

import { scrape, browserFetchURL } from "web-scrape";

Do not import server.js unless you intend to bind a port.

Status contract

Never treat non-ok bodies as page content. Important statuses:

status	meaning
`ok`	trustworthy extract
`cloudflare_blocked` / `turnstile_required`	CF/Turnstile gate (solve only if CapSolver env is set)
`captcha_required`	reCAPTCHA/hCaptcha (optional same-origin provider; off by default)
`robots_disallowed`	robots.txt denied
`blocked` / `login_required` / `rate_limited`	hard gates
`challenge_provider_unavailable`	CapSolver/provider failed
`timeout` / `error`	transport

Warnings may include turnstile:<code> — codes are secret-free. Never echo env secrets.

CapSolver rules

Opt-in: WEB_FETCH_TURNSTILE_PROVIDER=capsolver + WEB_FETCH_TURNSTILE_API_KEY.
Sticky proxy required for Cloudflare clearance (WEBSHARE_USERNAME_TEMPLATE={user}-{country}-1).
CapSolver proxy encoding is socks5; do not “fix” that to HTTP.
Do not send challenge-page html on AntiCloudflareTask — CapSolver returns invalid html.
Budget: hard CF clears often need 30–90s; set BROWSER_FETCH_TIMEOUT_MS ≥ 120000 when enabled.

Safety

Only public http(s) URLs (SSRF guard).
No social-login / cookie jar reuse — every browser context is ephemeral.
Do not disable robots checks.
Do not log API keys, proxy URLs, or solved tokens.

Read the full file on GitHub · 63 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 63 lines · 592 tokens per session scan A 4b1556b007b7

Subscribe to this mod's changes

web-scrape AGENTS.md is an instructions file published in the GitHub repository teslashibe/web-scrape (1 stars, last pushed 3mo ago), licensed Apache-2.0. It adds 592 tokens to every session, about $0.0024 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automation

web-scrape AGENTS.md next to 6 instructions from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
web-scrape AGENTS.mdteslashibe/web-scrape	Instructions file	1	3mo ago	Apache-2.0	A	592	Codex, OpenCode
vscode buildNext.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	9,719	GitHub Copilot
next.js AGENTS.mdvercel/next.js	Instructions file	143k	yesterday	MIT	A	7,461	Codex, OpenCode
langchain AGENTS.mdlangchain-ai/langchain	Instructions file	149k	yesterday	MIT	A	4,680	Codex, OpenCode
vscode oss-third-party-notices.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	5,516	GitHub Copilot
vscode copilot-instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	3,093	GitHub Copilot
deepseek-harness AGENTS.mddeepseek-ai/deepseek-harness	Instructions file	274k	2d ago	MIT	A	4,313	Codex, OpenCode

web-scrape: Instructions file for Codex