A command-line tool for scraping readable content and structured fields from web pages. It can also map a site's URLs, crawl multiple pages, and handle pages whose content is rendered by JavaScript.

Use it to extract page text or selected fields, crawl whole sites, find matching URLs, process single-page applications, or send extracted content to an AI model.

It removes repetitive manual copying from websites and provides consistent Markdown or JSON output for further processing.

webreaper — Skill for Claude Code and Codex by alex-on-ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add alex-on-ai/WebReaper --skill skill

Clone the repo

git clone --depth 1 https://github.com/alex-on-ai/WebReaper

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/alex-on-ai/webreaper/skill/github.svg)](https://agentmods.dev/skills/alex-on-ai/webreaper/skill)

Your own site

<a href="https://agentmods.dev/skills/alex-on-ai/webreaper/skill"><img src="https://agentmods.dev/badge/skills/alex-on-ai/webreaper/skill/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/alex-on-ai/webreaper/skill"><img src="https://agentmods.dev/badge/skills/alex-on-ai/webreaper/skill.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 413 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 3,075 The whole file, excluding the scripts and references it only reads on demand.

Security scan C 2 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00413	$0.03075
Opus 5.5	$0.00165	$0.01230
Sonnet 5.5	$0.00083	$0.00615
Haiku 4.5	$0.00041	$0.00308

Measured 1mo ago against content hash cddc6e6ee490, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade C, and why

webreaper scanned grade C with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Downloads and executes remote codehighSupply chain

curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.

`curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh`

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

`curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh`

WebReaper.Cli/skill/SKILL.md · 238 lines

How it starts

The opening of the file, as written. The whole thing — 238 lines — stays where its author put it; the contents beside it link to each section on GitHub.

WebReaper — scraping & extraction CLI

Three commands. Each is one shell call; output goes to stdout unless --output.

What to run

The user wants…	Run
The readable text of one page	`webreaper scrape <url>`
The readable text saved to a file	`webreaper scrape <url> --output page.md`
Specific fields from one page	`webreaper scrape <url> --schema schema.json`
Every page of a whole site (Markdown)	`webreaper crawl <url> > pages.jsonl`
Every page of a whole site (fields)	`webreaper crawl <url> --schema schema.json`
Fields with no schema (LLM, per page)	`webreaper scrape <url> --prompt "<what to extract>" --model <id> --llm-url <endpoint>`
Whole site, fields, cheaply (LLM)	`webreaper crawl <url> --infer "<goal>" --model <id> --llm-url <endpoint>`
One file per page (instead of JSONL)	add `--output-dir <dir>` to `scrape` / `crawl`
URLs on a site	`webreaper map <url>`
URLs matching a substring	`webreaper map <url> --search /blog/ --max-urls 50`
A JS-rendered page (SPA)	`webreaper scrape <url> --browser`
A bot-protected site	`webreaper scrape <url> --browser --auto-stealth`
Fields from every linked page	`webreaper scrape <index-url> --follow "<css selector>" --schema schema.json`

Whole-site crawl in one command

To cover an entire site, use crawl: it recursively follows every on-domain link from the start URL, extracts each page, and streams JSON Lines (one object per page). Markdown by default; --schema switches to field extraction.

# Every page, as Markdown, to a file.
webreaper crawl https://example.com > pages.jsonl

# Every page, specific fields, bounded.
webreaper crawl https://example.com --schema schema.json --max-pages 200

Flags: --max-pages <n> (default 1000), --max-depth <n> (hops from the start URL; default unlimited), --include-subdomains (default: same host + www only), --no-sitemap (skip sitemap seeding; recursive links only), --schema / --output (as for scrape). On-domain only; off-domain links are never followed. crawl starts at HTTP and auto-climbs to a browser on a blocked page (when one is installed), so a bot-protected site is handled; it does not render JS-only pages or use the stealth tier, so for a JS-rendered SPA or a site that needs stealth, scrape the pages with scrape --browser / scrape --stealth instead.

Read the full file on GitHub · 238 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago Changed · +2 lines cddc6e6ee490
2mo ago First seen · 236 lines · 413 tokens per session scan C 7e10f638b806

Subscribe to this mod's changes

webreaper is a skill published in the GitHub repository alex-on-ai/WebReaper (152 stars, last pushed 1mo ago), licensed MIT. It adds 413 tokens to every session and 3,075 once invoked, about $0.0017 per session on Opus 5.5. A static security scan graded it C with 2 findings (downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Browser automation Research crawler parser parsing scraper webcrawler webscraping

webreaper next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
webreaperalex-on-ai/WebReaper	Skill	152	1mo ago	MIT	C	413	Claude Code, Codex
firecrawl-buildfirecrawl/firecrawl	Skill	200k	today	AGPL-3.0	A	149	Claude Code, Codex
firecrawl-build-scrapefirecrawl/firecrawl	Skill	200k	today	AGPL-3.0	A	62	Claude Code, Codex
firecrawl-build-interactfirecrawl/firecrawl	Skill	200k	today	AGPL-3.0	A	56	Claude Code, Codex
crawl4ai-skilllancelin111/crawl4ai-skill	Skill	17	8mo ago	MIT	A	63	Claude Code, Codex
scrapling-officialD4Vinci/Scrapling	Skill	100k	today	BSD-3-Clause	A	76	Claude Code, Codex
firecrawl-build-onboardingfirecrawl/firecrawl	Skill	200k	today	AGPL-3.0	A	89	Claude Code, Codex

webreaper