Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/secondlifes/delphi-expert/rad-web-scrapingnpx skills add SecondLifes/delphi-expert --skill rad-web-scrapinggit clone --depth 1 https://github.com/SecondLifes/delphi-expertWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/secondlifes/delphi-expert/rad-web-scraping)<a href="https://agentmods.dev/skills/secondlifes/delphi-expert/rad-web-scraping"><img src="https://agentmods.dev/badge/skills/secondlifes/delphi-expert/rad-web-scraping.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00095 | $0.00977 |
| Opus 5 | $0.00048 | $0.00489 |
| Sonnet 5 | $0.00019 | $0.00195 |
| Haiku 4.5 | $0.00010 | $0.00098 |
Grade A, and why
rad-web-scraping scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
Copies of this mod
1 near-identical copy found in the catalogue:
- rad-web-scraping — 100% identical, 0 lines differ
How it starts
The opening of the file, as written. The whole thing — 39 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Web Scraping / Structured Data Extraction
When to Use
- Pulling structured data (prices, listings, articles, tables) from a website for a script, pipeline, or dataset.
- Turning arbitrary web pages into clean Markdown/text for an LLM, RAG index, or agent to consume.
- Deciding between a plain HTTP request, a scraping library, and full browser automation for a given target.
- Reviewing or debugging an existing scraper that's fragile or getting blocked.
Usage
| You say | What happens |
|---|---|
"Pull the prices from this site" / "Bu siteden ilan verilerini çek" |
Discovery-priority check first (API → embedded JSON → sitemap/feed → DOM as last resort, full order in references/discovery-and-extraction-patterns.md), then references/tool-selection.md's decision tree picks the library. |
"Turn these pages into clean Markdown for my RAG index" |
Recommends crawl4AI (LLM/RAG-ready output) per references/tool-selection.md. |
"My scraper keeps getting blocked / breaks randomly" |
Reviews against the retry/backoff/rate-limit baseline in references/discovery-and-extraction-patterns.md — a scraper missing these isn't hardened, it's incomplete. |
"Bypass this CAPTCHA / get around their bot detection" |
Refused — out of scope regardless of phrasing; reasonable retries/rate-limiting are fine, detection evasion is not. |
A scraping request with no target site/URL named ("Scrape it", "veriyi çek") |
Asks what site/data is meant before picking any tool — never guesses a target, and an empty request is never treated as "nothing to do." |
Golden Rules
- Discover before you parse. A REST/GraphQL API, a JSON blob embedded in the page's HTML, or a sitemap/feed is more stable than any CSS selector — always look for one before writing DOM-parsing code. Full priority order:
references/discovery-and-extraction-patterns.md. - Plain HTTP before a browser. Reach for
httpx/requestsfirst; only escalate to Playwright/browser automation when the target genuinely requires JS execution or session state a plain client can't replicate. A headless browser is slower, heavier, and more fragile than it looks. - Pick the tool for the job, not the first one you remember —
references/tool-selection.mdhas a verified decision tree (crawl4AI vs ScrapeGraphAI vs raw HTTP vs Playwright) with install commands, license, and when each wins. - Retry, back off, and rate-limit by default. A scraper with no retry/backoff either gets blocked or silently drops data the first time the network hiccups — this isn't optional hardening, it's baseline correctness for anything hitting a live site repeatedly.
- Scope and ethics — non-negotiable. Only scrape data you're legitimately allowed to access; respect
robots.txtand a site's terms of service. Never write code whose purpose is bypassing CAPTCHAs, defeating bot-detection, or evading paywalls/access controls — that's out of scope for this skill regardless of what's requested, matching this environment's own prohibition on bot-detection bypass. Reasonable technical robustness (retries, realistic headers, rate limiting) is fine; detection evasion is not.
What ships with it
2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 39 lines · 95 tokens per session scan A f0911fdc353e
rad-web-scraping is a skill published in the GitHub repository SecondLifes/delphi-expert (3 stars, last pushed 5d ago), licensed MIT. It adds 95 tokens to every session and 977 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
v8-jit
V8 JIT optimization patterns for writing high-performance JavaScript in Next.js server internals. Use when writing or reviewing hot-path code in app-render, stream-utils, routing, caching, or any per-request code path. Covers hidden classes / shapes, monomorphic call sites, inline caches, megamorphic deopt, closure…
use-agent-browser-for-airi
Test AIRI display-model imports with agent-browser across stage-tamagotchi Electron, stage-web, and stage-pocket mobile web layouts. Use when uploading and verifying contributor-supplied Live2D ZIP, VRM, or MMD ZIP/PMX/PMD files through AIRI's model selector, including onboarding bypass, format-specific import…
opencli-sitemap-author
Use when creating or maintaining OpenCLI site sitemaps: agent-facing navigation, page-state, action, workflow, API-reference, pitfall, and fallback knowledge for a website. Use after browser exploration discovers durable site context, when a sitemap is stale, or when promoting local site knowledge into the repo.
debug-optimize-lcp
Guides debugging and optimizing Largest Contentful Paint (LCP) using Chrome DevTools MCP tools. Use this skill whenever the user asks about LCP performance, slow page loads, Core Web Vitals optimization, or wants to understand why their page's main content takes too long to appear. Also use when the user mentions…
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
interactive-login
How to complete browser/interactive logins (aws / gh / glab / gcloud). The platform backgrounds the login poller so it survives the human's browser round-trip — and when that does NOT work.