ai-web-scraping-scrapegraph

ai-web-scraping-scrapegraph is a skill for Claude Code, Codex from gooseworks-ai/goose-skills. It costs 21 tokens per session (3,063 once invoked), scanned A, original, MIT.

An AI web-scraping service that extracts information from web pages and searches using plain-language instructions. It can also render JavaScript pages, crawl sites, read sitemaps, and turn pages into clean Markdown.

In plain words
What is it for?
Use it to extract selected data from web pages, search the web for information, crawl websites, collect sitemap URLs, retrieve raw HTML, or convert pages into readable Markdown.
Why use it?
It removes the need to write separate scraping rules for every site or manually copy information from pages. This is useful when important content is loaded by a browser or spread across many pages.

Skill for Claude CodeCodex

Written for no agent in particular: nothing here depends on one.

Good fit Use it to extract selected data from web pages, search the web for information, crawl websites, collect sitemap URLs, retrieve raw HTML, or convert pages into readable Markdown.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph
About the project

Goose Skills is a library of workflows and data APIs that lets coding agents handle growth and go-to-market work such as advertising, social media, content, SEO, lead generation, and customer research. It is intended for teams using Claude Code, Cursor, Codex, and similar agents. The catalogue entries are its reusable skills.

gooseworks-ai/goose-skills · 1,202 stars · on GitHub · gooseworks.ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add gooseworks-ai/goose-skills --skill ai-web-scraping-scrapegraph
Clone the repo
git clone --depth 1 https://github.com/gooseworks-ai/goose-skills

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for ai-web-scraping-scrapegraph

README.md
[![agentmods](https://agentmods.dev/badge/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph/github.svg)](https://agentmods.dev/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph)
Your own site
<a href="https://agentmods.dev/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph"><img src="https://agentmods.dev/badge/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for ai-web-scraping-scrapegraph

Your own site · 80×15
<a href="https://agentmods.dev/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph"><img src="https://agentmods.dev/badge/skills/gooseworks-ai/goose-skills/ai-web-scraping-scrapegraph.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 21 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 3,063 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector warn 7 Sept 2026
SkillSpector: 6 findings, up to high

These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →

  • high Privilege Escalation · line 12
    Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
    Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
  • high Privilege Escalation · line 14
    Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
    Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
  • high Privilege Escalation · line 15
    Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
    Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
  • high Privilege Escalation · line 18
    Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
    Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
  • medium MCP Rug Pull · line 18
    npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.
    Fix: Pin the version: npx @scope/[email protected]
  • medium Data Exfiltration · line 60
    Data is being sent to an external URL. This could be legitimate telemetry or data exfiltration. Manual review is recommended.
    Fix: Verify the destination URL is trusted and necessary. Remove or replace with documented APIs. Ensure no secrets, tokens, or PII are transmitted.
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00021 $0.03063
Opus 5 $0.00010 $0.01532
Sonnet 5 $0.00004 $0.00613
Haiku 4.5 $0.00002 $0.00306

Measured 9d ago against content hash a98d8a448278, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-12, from the pricing page.

Security

Grade A, and why

ai-web-scraping-scrapegraph scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
skills/research-tools/capabilities/ai-web-scraping-scrapegraph/SKILL.md · 233 lines

How it starts

The opening of the file, as written. The whole thing — 233 lines — stays where its author put it; the contents beside it link to each section on GitHub.

ScrapeGraph AI - Intelligent Web Scraping

Setup

Read your credentials from ~/.gooseworks/credentials.json:

export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")

If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login

All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"

Extract web content using AI with natural language prompts.

Capabilities

  • Start SmartScraper: Extract content from a webpage using AI by providing a natural language prompt and a URL
  • Start SearchScraper: Start a new AI-powered web search request
  • Scrape: Extract raw HTML content from web pages with JavaScript rendering support
  • Start SmartCrawler: Start a new web crawl request with AI extraction or markdown conversion
  • Start Sitemap: Extract all URLs from a website sitemap automatically
  • Start Markdownify: Convert any webpage into clean, readable Markdown format
  • Get SearchScraper Status: Get the status and results of a previous search request (free)
  • Get Markdownify Status: Check the status and retrieve results of a Markdownify request (free)
  • Get Sitemap Status: Check the status and retrieve results of a Sitemap request (free)
  • Get SmartCrawler Status: Get the status and results of a previous smartcrawl request (free)
  • Get SmartScraper Status: Check the status and retrieve results of a SmartScraper request (free)

Usage

Start SmartScraper

Extract content from a webpage using AI by providing a natural language prompt and a URL.

Parameters:

  • user_prompt* (string) - Natural language description of what information you want to extract from the webpage.
  • website_url* (string) - The URL of the webpage you want to extract information from. You must provide exactly one of: website_url, website_html, or website_markdown.
  • website_html (string) - Raw HTML content to process directly (max 2MB). Mutually exclusive with website_url and website_markdown. Useful when you already have HTML content cached or want to process modified HTML.
  • headers (object) - Optional custom HTTP headers to send with the request. Useful for setting User-Agent, cookies, authentication tokens, and other request metadata. Example: {"User-Agent": "Mozilla/5.0...", "Cookie": "session=abc123"}
  • output_schema (object) - Optional schema to structure the output. If provided, the AI will attempt to format the results according to this schema.
  • stealth (boolean) - Enable stealth mode to bypass bot protection using advanced anti-detection techniques. Adds +4 credits to the request cost
  • website_markdown (string) - Raw Markdown content to process directly (max 2MB). Mutually exclusive with website_url and website_html. Perfect for extracting structured data from Markdown documentation, README files, or any content already in Markdown format.
  • total_pages (number) - Optional parameter to enable pagination and scrape multiple pages. Specify the number of pages to extract data from. Default: 1 Range: 1-100
  • number_of_scrolls (number) - Optional parameter for infinite scroll pages. Specify how many times to scroll down to load more content before extraction. Default: 0 Range: 0-50
  • render_heavy_js (boolean) - Optional parameter to enable enhanced JavaScript rendering for heavy JS websites (React, Vue, Angular, SPAs). Use when standard rendering doesn’t capture all content. Default: false
  • mock (boolean) - Optional parameter to enable mock mode. When set to true, the request will return mock data instead of performing an actual extraction. Useful for testing and development. Default: false
  • cookies (object) - Optional cookies object for authentication and session management. Useful for accessing authenticated pages or maintaining session state. Example: {"session_id": "abc123", "auth_token": "xyz789"}
  • steps (array) - Optional array of interaction steps to perform on the webpage before extraction. Each step is a string describing the action to take (e.g., “click on filter button”, “wait for results to load”). Example: ["click on search button", "type query in search box", "wait for results"]

Read the full file on GitHub · 233 lines

Files

What ships with it

1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 9d ago First seen · 233 lines · 21 tokens per session scan A a98d8a448278

Subscribe to this mod's changes

ai-web-scraping-scrapegraph is a skill published in the GitHub repository gooseworks-ai/goose-skills (1,202 stars, last pushed 11d ago), licensed MIT. It adds 21 tokens to every session and 3,063 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.

Related

Other skills, from other repositories

parallel-web

Use Parallel CLI for web search, URL extraction, deep research, structured data enrichment, entity discovery, and recurring web monitoring. Best for requests that explicitly need current web evidence, academic-source discovery, repeated entity lookups, exhaustive reports, or ongoing change tracking.

K-Dense-AI/scientific-agent-skills · 55 tokens

agent-browser

Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.

code-yeongyu/oh-my-openagent · 51 tokens

ultimate-browsing

Reaches web pages a plain fetch cannot: renders JS, drives clicks and forms, captures screenshots, holds a login, and gets past WAF blocks through platform-native readers and stealth Chrome. Use for any page work beyond retrieving static text.

code-yeongyu/oh-my-openagent · 53 tokens

dev-browser

Browser automation with persistent page state. Use when users ask to navigate websites, fill forms, take screenshots, extract web data, test web apps, or automate browser workflows. Trigger phrases include "go to [url]", "click on", "fill out the form", "take a screenshot", "scrape", "automate", "test the website"…

code-yeongyu/oh-my-openagent · 84 tokens

devlab-web-visual-ops

A browser-based operations guide for web applications, including single-page apps, micro-frontends, and multi-page sites. It covers observing, navigating, interacting, inspecting, validating, and capturing pages with browser automation and visual fallbacks.

seed-forge/harness-ai-kit · 176 tokens

read-deleted-pages

Recover deleted, edited or historical web content using the Wayback Machine and its CDX API, archive.today, Common Crawl and Memento/Timetravel. Use when a page is deleted, changed or 404s, checking what a site used to say, finding old team or staff pages, prior pricing, removed posts, pre-redaction wording or old…

UseOSINT/Skills · 131 tokens