web-scraping

Web Scraping Patterns

Patterns for extracting structured data from web pages, including pagination, SPAs, authenticated scraping, and rate-limited crawls.

1. Core Scraping Loop

The fundamental extract-one-page workflow:

1. Navigate to page
   mcp__browser-use__browser_navigate(url="https://shop.example.com/products")
   → session_id: "s1"

2. Get DOM state (find data containers and pagination controls)
   mcp__browser-use__browser_get_state(session_id="s1")
   → selector_map: {... "47": {tag: "a", text: "Next →", href: "..."}, ...}

3. Extract structured data with LLM-powered query
   mcp__browser-use__browser_extract_content(
     query="product name, price, SKU, availability for each product listed",
     session_id="s1"
   )
   → "Product: Widget A, Price: $12.99, SKU: WA-001, In Stock\n..."

4. Append to results array

5. Check pagination (see Section 2)

6. Close session when done
   mcp__browser-use__browser_close_session(session_id="s1")

7. Write results to file
   Write tool → products.json

When to Use extract_content vs get_html

2. Pagination Patterns

2.1 Numbered Page Navigation

For sites with ?page=N URL patterns:

base_url = "https://shop.example.com/products"
all_results = []

for page in range(1, max_pages + 1):
    url = f"{base_url}?page={page}"
    mcp__browser-use__browser_navigate(url=url, session_id=session_id)
    state = mcp__browser-use__browser_get_state(session_id=session_id)

    # Check if page has content (stop if we reach an empty page)
    if "No products found" in str(state["selector_map"]):
        break

    data = mcp__browser-use__browser_extract_content(
        query="all product names, prices, and URLs",
        session_id=session_id
    )
    all_results.append(data["content"])

    # Rate limit: pause between pages
    Bash: sleep 1