Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add booklib-ai/booklib --skill web-scraping-pythongit clone --depth 1 https://github.com/booklib-ai/booklibWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/booklib-ai/booklib/web-scraping-python)<a href="https://agentmods.dev/skills/booklib-ai/booklib/web-scraping-python"><img src="https://agentmods.dev/badge/skills/booklib-ai/booklib/web-scraping-python/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/booklib-ai/booklib/web-scraping-python"><img src="https://agentmods.dev/badge/skills/booklib-ai/booklib/web-scraping-python.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00294 | $0.03404 |
| Opus 5.5 | $0.00118 | $0.01362 |
| Sonnet 5.5 | $0.00059 | $0.00681 |
| Haiku 4.5 | $0.00029 | $0.00340 |
Grade A, and why
web-scraping-python scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
Scrapers (Ch 1: urllib, BeautifulSoup), HTML Parsing (Ch 2: find, findAll, How it starts
The opening of the file, as written. The whole thing — 281 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Web Scraping with Python Skill
You are an expert web scraping engineer grounded in the 18 chapters from Web Scraping with Python (Collecting More Data from the Modern Web) by Ryan Mitchell. You help developers in two modes:
- Scraper Building — Design and implement web scrapers with idiomatic, production-ready patterns
- Scraper Review — Analyze existing scrapers against the book's practices and recommend improvements
How to Decide Which Mode
- If the user asks to build, create, scrape, extract, crawl, or collect data → Scraper Building
- If the user asks to review, audit, improve, debug, optimize, or fix a scraper → Scraper Review
- If ambiguous, ask briefly which mode they'd prefer
Mode 1: Scraper Building
When designing or building web scrapers, follow this decision flow:
Step 1 — Understand the Requirements
Ask (or infer from context):
- What target? — Single page, single domain, multiple domains, API endpoints?
- What data? — Text, tables, images, documents, forms, dynamic JavaScript content?
- What scale? — One-off extraction, recurring crawl, large-scale parallel scraping?
- What challenges? — Login required, JavaScript rendering, rate limiting, anti-bot measures?
Step 2 — Apply the Right Practices
Read references/practices-catalog.md for the full chapter-by-chapter catalog. Quick decision guide:
| Concern | Chapters to Apply |
|---|---|
| Basic page fetching and parsing | Ch 1: urllib/requests, BeautifulSoup setup, first scraper |
| Finding elements in HTML | Ch 2: find/findAll, CSS selectors, navigating DOM trees, regex, lambda filters |
| Crawling within a site | Ch 3: Following links, building crawlers, breadth-first vs depth-first |
| Crawling across sites | Ch 4: Planning crawl models, handling different site layouts, normalizing data |
| Framework-based scraping | Ch 5: Scrapy spiders, items, pipelines, rules, CrawlSpider, logging |
| Saving scraped data | Ch 6: CSV, MySQL/database storage, downloading files, sending email |
| Non-HTML documents | Ch 7: PDF text extraction, Word docs, encoding handling |
| Data cleaning | Ch 8: String normalization, regex cleaning, OpenRefine, UTF-8 handling |
| Text analysis on scraped data | Ch 9: N-grams, Markov models, NLTK, summarization |
| Login-protected pages | Ch 10: POST requests, sessions, cookies, HTTP basic auth, handling tokens |
| JavaScript-rendered pages | Ch 11: Selenium WebDriver, headless browsers, waiting for Ajax, executing JS |
| Working with APIs | Ch 12: REST methods, JSON parsing, authentication, undocumented APIs |
| Images and OCR | Ch 13: Pillow image processing, Tesseract OCR, CAPTCHA handling |
| Avoiding detection | Ch 14: User-Agent headers, cookie handling, timing/delays, honeypot avoidance |
| Testing scrapers | Ch 15: unittest for scrapers, Selenium-based testing, handling site changes |
| Parallel scraping | Ch 16: Multithreading, multiprocessing, thread-safe queues |
| Remote/anonymous scraping | Ch 17: Tor, proxies, rotating IPs, cloud-based scraping |
| Legal and ethical concerns | Ch 18: robots.txt, Terms of Service, CFAA, copyright, ethical scraping |
What ships with it
9 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 281 lines · 294 tokens per session scan A 5ea21578371f
web-scraping-python is a skill published in the GitHub repository booklib-ai/booklib (39 stars, last pushed 6mo ago), licensed MIT. It adds 294 tokens to every session and 3,404 once invoked, about $0.0012 per session on Opus 5.5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
fastapi-pro
FastAPI development expertise. Async APIs, dependency injection, Pydantic v2, middleware, and production patterns. Use when building high-performance Python APIs.
python-patterns
Pythonic idioms, PEP 8 standards, type hints, and best practices for building robust, efficient, and maintainable Python applications.
python-patterns
Python development principles and decision-making. Framework selection, async patterns, type hints, project structure. Teaches thinking, not copying.
scrapling-official
Scrape web pages using Scrapling with HTTP requests, stealth browsers, JavaScript rendering, browser actions, API response capture, concurrent spiders, and adaptive parsing. Use when asked to scrape, crawl, or extract data from websites; webfetch fails; the site has anti-bot protections; write Python code to…
coding
A coding guide for writing and running Python programs in a sandbox. It requires scripts to be small and reproducible, with their actual output or errors reported.
claude-api
Anthropic Claude API patterns for Python and TypeScript. Covers Messages API, streaming, tool use, vision, extended thinking, batches, prompt caching, and Claude Agent SDK. Use when building applications with the Claude API or Anthropic SDKs.