Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add yale-som-hpc/claude-code-marketplace --skill scraping-at-scalegit clone --depth 1 https://github.com/yale-som-hpc/claude-code-marketplaceWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/yale-som-hpc/claude-code-marketplace/scraping-at-scale)<a href="https://agentmods.dev/skills/yale-som-hpc/claude-code-marketplace/scraping-at-scale"><img src="https://agentmods.dev/badge/skills/yale-som-hpc/claude-code-marketplace/scraping-at-scale/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/yale-som-hpc/claude-code-marketplace/scraping-at-scale"><img src="https://agentmods.dev/badge/skills/yale-som-hpc/claude-code-marketplace/scraping-at-scale.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00082 | $0.03235 |
| Opus 5 | $0.00041 | $0.01618 |
| Sonnet 5 | $0.00016 | $0.00647 |
| Haiku 4.5 | $0.00008 | $0.00324 |
Grade C, and why
scraping-at-scale scanned grade C with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Recursive force deletehighDestructive command
rm -rf with a variable or a broad path is one typo away from removing the wrong tree.
trap 'rm -rf "$workdir"' EXIT Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
body, headers = fetch(url) # your HTTP layer (see acquiring-data) How it starts
The opening of the file, as written. The whole thing — 232 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Scraping at Scale
Rule: for a large crawl, separate the catalog (what's cached) from the bodies (the data) from the action log (what the run did). Make the catalog durable on GPFS so the job is resumable, and never materialize a million loose files.
This is the heavy machinery for crawls of tens of thousands of pages or more. For the common case (WRDS, a few API pulls, credentials, a request-hash cache), use acquiring data instead — most data work never needs what's here.
Three separate stores
data/raw_html/<aa>/<key>.html # bodies, sharded by 2-char hash prefix
data/raw_json/<aa>/<key>.json
data/derived/ # parsed outputs
data/metadata.db # catalog: SQLite, one row per stored artifact
data/fetch_log.jsonl # optional: JSONL, one row per fetch attempt
Save bodies under raw, parse separately into derived — if parsing changes, re-parse without re-fetching. Then keep two records that answer different questions:
- Catalog (
metadata.db, SQLite). One canonical row per stored artifact, keyed bykey:url,final_url,status,content_type,bytes,etag,last_modified,fetched_at. UPSERT on each success — a 304 revalidation just updateslast_modified/fetched_atwithout duplicate rows. Answers "what's in the cache?" - Action log (
fetch_log.jsonl, optional). Append-only, one row per attempt:ts,url,attempt,outcome(ok/cache_hit/retry/error),status,key,error. Answers "what did the scraper do this run?" — including failures that produced no body.
Different shapes, different formats: the catalog has one-row-per-key identity, lookup, and updates (SQLite); the log is append-only and read as a stream (JSONL, safe to multi-write under O_APPEND).
Use WAL for the catalog. SQLite's default journal (DELETE) serializes readers and writers — a DuckDB query during a scrape blocks the next upsert. WAL gives concurrent reads + serialized writes, halves per-commit fsync, and with synchronous = NORMAL is durable (a crash loses at most the last in-flight transaction, never corrupts). The helper below sets it up.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 232 lines · 82 tokens per session scan C f6c71963f7c5
scraping-at-scale is a skill published in the GitHub repository yale-som-hpc/claude-code-marketplace (5 stars, last pushed 2mo ago), licensed Unlicense. It adds 82 tokens to every session and 3,235 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it C with 2 findings (recursive force delete, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
create-pr
Creates a GitHub PR with a Linear-ticket-prefixed title and a decision-led, narrative description for prisma-next. Use when the user wants to create a pull request, open a PR, or submit changes for review.
schema-exploration
Lists tables, describes columns and data types, identifies foreign key relationships, and maps entity relationships in a database. Use when the user asks about database schema, table structure, column types, what tables exist, ERD, foreign keys, or how entities relate.
ha-data-stores
Map of Hope Agent's local data stores and safe read-only query workflow. Use when the user asks where Hope Agent stores data, wants to inspect sessions/messages/memory/logs/background jobs/knowledge indexes/settings, asks the model to query local app data, or debugging requires checking persisted state. Trigger…
supabase
Supabase / PostgREST Row-Level-Security playbook — pull the anon (or leaked servicerole) key out of the frontend JS, map tables from the auto-generated OpenAPI spec, test anonymous RLS READ disclosures (PII/secret leaks), and anonymous RLS WRITE abuse (insert/update/delete — e.g. forging…
nornicdb-cypher-queries
Pick fast, predictable Cypher query shapes in NornicDB — point lookups, batch retrieval, pagination, search, traversal, batched UNWIND/MERGE writes, cleanup, multi-tenant isolation. Use when writing or reviewing Cypher whose latency or throughput matters; maps user intent to the executor's hot-path query templates.
dsql
Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, diagnose cluster performance, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, foreign key…