web-scraping

Web Scraping Skill

Toolkit for efficient web content extraction using the scraper MCP server tools.

When to Use This Skill

• Extracting content from web pages for analysis
• Converting web pages to markdown for LLM consumption
• Extracting plain text from HTML documents
• Harvesting links from web pages
• Batch processing multiple URLs concurrently

Available Tools

Tool	Purpose	Best For
mcp__scraper__scrape_url	Convert HTML to markdown	LLM-friendly content extraction
mcp__scraper__scrape_url_html	Raw HTML content	DOM inspection, metadata extraction
mcp__scraper__scrape_url_text	Plain text extraction	Clean text without formatting
mcp__scraper__scrape_extract_links	Link harvesting	Site mapping, crawling

Tool Usage

1. Markdown Conversion (Recommended for LLMs)

Convert web pages to clean markdown format:

mcp__scraper__scrape_url(
    urls=["https://example.com/article"],
    css_selector=".article-content",
    timeout=30,
    max_retries=3
)

Response includes:

• content: Markdown-formatted text
• url: Final URL (after redirects)
• status_code: HTTP status
• metadata: Headers, timing, retry info

2. Raw HTML Extraction

Get unprocessed HTML for DOM analysis:

mcp__scraper__scrape_url_html(
    urls=["https://example.com"],
    css_selector="meta",
    timeout=30
)

Use cases:

• Extracting meta tags and Open Graph data
• Inspecting page structure
• Getting specific HTML elements

3. Plain Text Extraction

Extract readable text without HTML markup:

mcp__scraper__scrape_url_text(
    urls=["https://example.com/page"],
    strip_tags=["script", "style", "nav", "footer"],
    css_selector="#main-content"
)

Parameters:

• strip_tags: HTML elements to remove before extraction (default: script, style, meta, link, noscript)

4. Link Extraction

Harvest all links from a page:

mcp__scraper__scrape_extract_links(
    urls=["https://example.com"],
    css_selector="nav.primary"
)