Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add moonlight-lupin/agent-skills --skill website-scrapinggit clone --depth 1 https://github.com/moonlight-lupin/agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/moonlight-lupin/agent-skills/website-scraping)<a href="https://agentmods.dev/skills/moonlight-lupin/agent-skills/website-scraping"><img src="https://agentmods.dev/badge/skills/moonlight-lupin/agent-skills/website-scraping/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/moonlight-lupin/agent-skills/website-scraping"><img src="https://agentmods.dev/badge/skills/moonlight-lupin/agent-skills/website-scraping.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00210 | $0.05537 |
| Opus 5 | $0.00105 | $0.02769 |
| Sonnet 5 | $0.00042 | $0.01107 |
| Haiku 4.5 | $0.00021 | $0.00554 |
Grade A, and why
website-scraping scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
**2.0b — Is this a hostile mega-platform?** A handful of sites — Amazon, LinkedIn, Instagram, TikTok, Facebook, YouTube, Zillow, Google Maps, Crunchbase, and similar — invest heavily in defeating DIY scraping and change How it starts
The opening of the file, as written. The whole thing — 263 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Website scraping
A workflow for turning websites into structured data. Bias toward the lightest tool that works: static HTML → JSON-in-page → reverse-engineered API → browser automation, in that order. Skipping recon and reaching straight for Playwright is the single most common way to waste hours.
When to use
This skill applies to any task shaped like "given a URL or a list of URLs, produce structured records". Examples:
- Competitor price monitoring (hotels, flights, retail, rentals, student housing, SaaS)
- E-commerce product catalogue extraction
- Real-estate or job-listing harvesting
- Reviews, ratings, availability calendars
- One-off "pull this table into a spreadsheet" jobs
Out of scope: database persistence, scheduling, dashboards, downstream normalisation against a fixed schema. This skill produces JSONL — what the caller does with it is their problem.
Workflow
Follow these steps in order. Most failures come from skipping step 1.
1. Frame the goal before touching code
Pin down the answers in writing before opening an editor:
- What records do you want? One row per what? Per product? Per (product × variant × date)? Per (hotel × room-type × check-in)? Be explicit — many sites force you to choose between "one row per listing card" and "one row per price-variant", and the choice cascades through the whole scraper.
- What fields per record? Required vs nice-to-have. Mark anything the source might not publish on every record (sometimes-missing fields are normal, not a bug).
- Input shape: a single URL? A list of URLs? A search query you need to execute first? A seed listing page you crawl from?
- Output shape: this skill recommends JSONL (one JSON object per line, caller-defined keys) plus a sidecar manifest file. See "Output convention" below.
- Scale & cadence: how many records, how often? This one answer drives every downstream cost decision. A one-off harvest of 100 pages and a daily refresh of 10,000 SKUs are different projects: the first just needs a polite
time.sleep; the second has to justify proxy/managed-service economics (see step 4, Tier 5+). Pin down volume and repeat-frequency now, not after you've built a single-shot scraper that can't keep up. - What's "ground truth"? Pick one specific item you'll eyeball in the browser end-to-end to verify the scraper matches.
What ships with it
23 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- .gitignore 175 B
- evals/evals.json 1.6 KB
- evals/routing-fixtures.json 2.6 KB
- examples/agentic-browsing-quotes-demo/explore.py 2.1 KB runs code
- examples/agentic-browsing-quotes-demo/final_runs/run_1/final_script_log.txt 432 B
- examples/agentic-browsing-quotes-demo/final_runs/run_1/final_script.py 3.4 KB runs code
- examples/agentic-browsing-quotes-demo/final_runs/run_1/records.jsonl 261 B
- examples/agentic-browsing-quotes-demo/final_runs/run_1/screenshots/final_execution_1_author_selected.png 26 KB
- examples/agentic-browsing-quotes-demo/final_runs/run_1/screenshots/final_execution_2_tag_selected.png 27 KB
- examples/agentic-browsing-quotes-demo/final_runs/run_1/screenshots/final_execution_3_results.png 35 KB
- examples/agentic-browsing-quotes-demo/plan.md 609 B
- examples/agentic-browsing-quotes-demo/README.md 2.7 KB
- examples/agentic-browsing-quotes-demo/screenshots/explore_1_start.png 25 KB
- examples/agentic-browsing-quotes-demo/screenshots/explore_2_after_author.png 26 KB
- README.md 697 B
- references/agentic-browsing.md 9.7 KB
- references/anti-bot.md 13 KB
- references/extraction.md 14 KB
- references/named-div-extraction.md 5.3 KB
- references/playwright.md 17 KB
- references/recon.md 18 KB
- references/scrapling.md 6.0 KB
- scripts/recon.py 15 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 263 lines · 210 tokens per session scan A 3864eb5b0291
website-scraping is a skill published in the GitHub repository moonlight-lupin/agent-skills (62 stars, last pushed 4d ago), licensed MIT. It adds 210 tokens to every session and 5,537 once invoked, about $0.0011 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
firecrawl
Scrape, search et interagit avec le web via Firecrawl — extrait du contenu propre en Markdown depuis n'importe quelle URL, recherche sur le web, crawl de sites entiers, et interaction avec des pages dynamiques (clics, formulaires). Clef API dans FIRECRAWLAPIKEY.
playwright-cli
Automate browser interactions, test web pages and work with Playwright tests.
hive.slack-notifications-setup
Set up a Slack notification channel (Sentinel) for a colony by driving the browser — reuse or create the "Hive Sentinel" Slack app from a JSON manifest, install it, capture the bot + app tokens, create/select the channel via the Slack API, and turn Sentinel on so the colony can ping the user on Slack and accept…
browser-edge-cases
SOP for debugging browser automation failures on complex websites. Use when browser tools fail on specific sites like LinkedIn, Twitter/X, SPAs, or sites with Shadow DOM.
amazon-search-listing
Amazon search and category listing scraper: extract product listings from any Amazon search results page, keyword search URL, or category browse page and return per-item cards (asin, title, url, image, price, listPrice, stars, reviewCount, badges, isAmazonChoice, isBestSeller, isSponsored, delivery, boughtInPast…
walmart-product-detail
Walmart product detail page extractor: given a walmart.com product URL (walmart.com/ip/...), extract full product data including itemId, title, brand, model, UPC, price, wasPrice, currency, availability, category path, seller info, all images, shortDescription, longDescription, product highlights, full specifications…