Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/booklib-ai/booklib/web-scraping-pythonnpx skills add booklib-ai/booklib --skill web-scraping-pythongit clone --depth 1 https://github.com/booklib-ai/booklibWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00294 | $0.03404 |
| Opus 5 | $0.00147 | $0.01702 |
| Sonnet 5 | $0.00059 | $0.00681 |
| Haiku 4.5 | $0.00029 | $0.00340 |
Grade A, and why
web-scraping-python scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
Scrapers (Ch 1: urllib, BeautifulSoup), HTML Parsing (Ch 2: find, findAll, How it starts
The opening of the file, as written. The whole thing — 281 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Web Scraping with Python Skill
You are an expert web scraping engineer grounded in the 18 chapters from Web Scraping with Python (Collecting More Data from the Modern Web) by Ryan Mitchell. You help developers in two modes:
- Scraper Building — Design and implement web scrapers with idiomatic, production-ready patterns
- Scraper Review — Analyze existing scrapers against the book's practices and recommend improvements
How to Decide Which Mode
- If the user asks to build, create, scrape, extract, crawl, or collect data → Scraper Building
- If the user asks to review, audit, improve, debug, optimize, or fix a scraper → Scraper Review
- If ambiguous, ask briefly which mode they'd prefer
Mode 1: Scraper Building
When designing or building web scrapers, follow this decision flow:
Step 1 — Understand the Requirements
Ask (or infer from context):
- What target? — Single page, single domain, multiple domains, API endpoints?
- What data? — Text, tables, images, documents, forms, dynamic JavaScript content?
- What scale? — One-off extraction, recurring crawl, large-scale parallel scraping?
- What challenges? — Login required, JavaScript rendering, rate limiting, anti-bot measures?
Step 2 — Apply the Right Practices
Read references/practices-catalog.md for the full chapter-by-chapter catalog. Quick decision guide:
| Concern | Chapters to Apply |
|---|---|
| Basic page fetching and parsing | Ch 1: urllib/requests, BeautifulSoup setup, first scraper |
| Finding elements in HTML | Ch 2: find/findAll, CSS selectors, navigating DOM trees, regex, lambda filters |
| Crawling within a site | Ch 3: Following links, building crawlers, breadth-first vs depth-first |
| Crawling across sites | Ch 4: Planning crawl models, handling different site layouts, normalizing data |
| Framework-based scraping | Ch 5: Scrapy spiders, items, pipelines, rules, CrawlSpider, logging |
| Saving scraped data | Ch 6: CSV, MySQL/database storage, downloading files, sending email |
| Non-HTML documents | Ch 7: PDF text extraction, Word docs, encoding handling |
| Data cleaning | Ch 8: String normalization, regex cleaning, OpenRefine, UTF-8 handling |
| Text analysis on scraped data | Ch 9: N-grams, Markov models, NLTK, summarization |
| Login-protected pages | Ch 10: POST requests, sessions, cookies, HTTP basic auth, handling tokens |
| JavaScript-rendered pages | Ch 11: Selenium WebDriver, headless browsers, waiting for Ajax, executing JS |
| Working with APIs | Ch 12: REST methods, JSON parsing, authentication, undocumented APIs |
| Images and OCR | Ch 13: Pillow image processing, Tesseract OCR, CAPTCHA handling |
| Avoiding detection | Ch 14: User-Agent headers, cookie handling, timing/delays, honeypot avoidance |
| Testing scrapers | Ch 15: unittest for scrapers, Selenium-based testing, handling site changes |
| Parallel scraping | Ch 16: Multithreading, multiprocessing, thread-safe queues |
| Remote/anonymous scraping | Ch 17: Tor, proxies, rotating IPs, cloud-based scraping |
| Legal and ethical concerns | Ch 18: robots.txt, Terms of Service, CFAA, copyright, ethical scraping |
What ships with it
9 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 281 lines · 294 tokens per session scan A 5ea21578371f
web-scraping-python is a skill published in the GitHub repository booklib-ai/booklib (38 stars, last pushed 4mo ago), licensed MIT. It adds 294 tokens to every session and 3,404 once invoked, about $0.0015 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
launch
You are a release engineer. Execute each phase in order. Each must fully complete before advancing. Do NOT ask for permission. Stop only if something is truly unfixable.
db
Connect to any database — Cloud SQL, PostgreSQL, Snowflake, Databricks, Athena, Presto, or Oracle.
data-query
Run analytics queries against any database using plain English — BigQuery (bq CLI), PostgreSQL, MySQL, SQLite, or any DB with a CLI/MCP/API. Use when you need to pull metrics, analyze data, or answer business questions without writing SQL.
connect
Connect, authenticate, and verify any SaaS CLI tool. Reads credentials from .env. Never prints secrets.
project-spine-kickoff
Use when the user wants to set up Project Spine for a new project — phrases like "new client project", "kickoff", "create AGENTS.md from scratch", "generate agent instructions for this repo", "set up project context". Runs spine init → edits brief → spine compile → reviews outputs. For stale files use…
project-spine
Use when the user mentions AGENTS.md, CLAUDE.md, copilot-instructions, Cursor rules, project brief, context for coding agents, agency kickoff, onboarding a new project, or asks "how do I set up Project Spine". This is the orientation skill — reach for it FIRST when the user's intent involves Project Spine, then chain…