Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/ScrapeOps/scrapeops-scraping-assistant-claude-pluginnpx agentmods add skills/scrapeops/scrapeops-scraping-assistant-claude-plugin/generate-crawler-scraperWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/scrapeops/scrapeops-scraping-assistant-claude-plugin/generate-crawler-scraper)<a href="https://agentmods.dev/skills/scrapeops/scrapeops-scraping-assistant-claude-plugin/generate-crawler-scraper"><img src="https://agentmods.dev/badge/skills/scrapeops/scrapeops-scraping-assistant-claude-plugin/generate-crawler-scraper.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00119 | $0.27360 |
| Opus 5 | $0.00060 | $0.13680 |
| Sonnet 5 | $0.00024 | $0.05472 |
| Haiku 4.5 | $0.00012 | $0.02736 |
Grade B, and why
generate-crawler-scraper scanned grade B with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Reads agent configuration directoriesmediumAgent snooping
.claude/, .codex/, .gemini/ hold keys, settings and other credentials a mod has no legitimate need for.
> **API key handling**: The MCP server reads `SCRAPEOPS_API_KEY` from `~/.claude/settings.json` automatically. If a tool call fails with "SCRAPEOPS_API_KEY is not configured", ask the user for their key, write it to `~/. Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
from urllib.parse import urlsplit, parse_qsl How it starts
The opening of the file, as written. The whole thing — 2,238 lines — stays where its author put it; the contents beside it link to each section on GitHub.
ScrapeOps — Generate Crawler + Scraper Skill (local-orchestrated)
Build a runnable crawler+scraper pair locally. Two files are produced by the Go backend (one per phase) and you run them on the user's machine. Each generated script talks to the ScrapeOps proxy directly (no server-side orchestration).
Default layout is SEPARATE: crawler writes URLs to a JSONL file, scraper reads that file and produces product JSONL. No merged script.
Core rule: the Go backend ONLY generates code. Everything else — URL discovery, schema generation, local execution, JSONL handoff — happens here in the Claude Code session.
API key handling: The MCP server reads
SCRAPEOPS_API_KEYfrom~/.claude/settings.jsonautomatically. If a tool call fails with "SCRAPEOPS_API_KEY is not configured", ask the user for their key, write it to~/.claude/settings.jsonunderenv.SCRAPEOPS_API_KEY, and retry.
Step 1 — Collect inputs
Ask for each value unless the user stated it in their opening message. Never assume language/library.
| Slot | Description |
|---|---|
domain |
Target site, e.g. amazon.com, walmart.com, mercadolivre.com.br |
search_query |
Natural-language search term, e.g. "mens shirts", "camisetas masculinas" |
language |
Ask. Python, JavaScript, PHP, Ruby, Go, Rust, Java, or C# |
library |
Ask. Depends on language (BeautifulSoup, Cheerio, symfony/dom-crawler, nokogiri, goquery, scraper, jsoup, HtmlAgilityPack) |
max_pages |
Pagination cap. Default 1 — ask via the fixed AskUserQuestion format in Step 1.4 |
1.1 — Domain and query
Extract from the user's message. "Crawler da Amazon buscando camisetas" → amazon.com, camisetas. If either is ambiguous, ask.
1.2 — Language (obligatory question if not provided)
Use AskUserQuestion:
question: "Which language do you want the crawler + scraper in?"
header: "Language"
options:
- label: "Python"
- label: "JavaScript"
- label: "PHP"
- label: "Other"
description: "Ruby, Go, Rust, Java, C#"
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 2,238 lines · 119 tokens per session scan B e3f21a6ad045
generate-crawler-scraper is a skill published in the GitHub repository ScrapeOps/scrapeops-scraping-assistant-claude-plugin (5 stars, last pushed 4mo ago), licensed MIT. It adds 119 tokens to every session and 27,360 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it B with 2 findings (reads agent configuration directories, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ebay-search-listing
Extracts product listings from any eBay search or category page URL, returning per-item cards (itemNumber, url, title, subtitle, caption, price, priceWithCurrency, currency, wasPrice, bids, shipping, seller, sellerFeedbackCount, sellerPositiveRating, reviewsCount, starRating, image) plus pagination state (currentPage…
amazon-bestseller-listing
Amazon Best Sellers listing scraper: extract product cards from any Amazon Best Sellers (zgbs) or /gp/bestsellers/ category page — returns rank (position on chart), asin, title, url, image, imageAlt, price, stars, reviewCount, ratingRaw per item, plus category metadata (categoryName, categoryFullName, categoryUrl) and…
amazon-search-listing
Amazon search and category listing scraper: extract product listings from any Amazon search results page, keyword search URL, or category browse page and return per-item cards (asin, title, url, image, price, listPrice, stars, reviewCount, badges, isAmazonChoice, isBestSeller, isSponsored, delivery, boughtInPast…
etsy-product-detail
Etsy product detail scraper: given an Etsy listing URL, returns full product detail including listingId, title, priceCurrent, priceOriginal, currency, images (all), description, shopName, shopUrl, rating, reviewCount, favorites, inCartCount, variations (with per-option price ranges), highlights, listedDate…
walmart-product-detail
Walmart product detail page extractor: given a walmart.com product URL (walmart.com/ip/...), extract full product data including itemId, title, brand, model, UPC, price, wasPrice, currency, availability, category path, seller info, all images, shortDescription, longDescription, product highlights, full specifications…
ecommerce-listing
Extract product list from any e-commerce category page, search results page, or keyword search with filters. Returns paginated product arrays with URL, name, price, currency, image, rating, review count per item. Supports URL input, keyword search, and site-scoped search with filters: price range, brand, category…