structured-extraction

Structured web data extraction skill.

When to activate

Use this skill when you need to:

• Extract specific data points from a web page (prices, features, specs)
• Build a comparison table from multiple pages or sites
• Scrape a list of items from a page (search results, product listings)
• Extract tabular data from a web page into a structured format

Methodology

1. Plan the extraction

Use the think tool to identify:

• What data points to extract (columns in your target table)
• Which pages contain the data (URLs or navigation paths)
• Whether the data is on one page or spread across multiple pages
• Whether pagination or interaction is needed to reveal the data

2. Navigate to the data

Open the target URL and confirm you landed on the right page:

open_url("https://example.com/pricing")

Check the title and URL in the response to verify.

3. Snapshot the page

Take a snapshot to understand the page structure:

snapshot()

Look for:

• Data containers (tables, cards, lists)
• Interactive elements that reveal more data (tabs, accordions, "Show more" buttons)
• Pagination controls

If the page has distinct sections, use a CSS selector to scope the snapshot: snapshot(selector=".pricing-table")

4. Extract text content

Use get_text to pull text from specific elements or the full page:

get_text(ref="@e5")   # specific element
get_text()            # full page text

For tabular data, extracting the full page text often captures tables in a readable format.

5. Take evidence screenshots

Screenshot key pages for the research report:

screenshot(full_page=false)           # viewport
screenshot(full_page=true)            # full page
screenshot(annotate=true)             # with element labels

Include screenshot paths in your report so the user can review the raw source.

6. Process with Python

Use Python to structure the extracted text into clean data:

data = [
    {"name": "Plan A", "price": "$10/mo", "features": "5 users, 10GB"},
    {"name": "Plan B", "price": "$25/mo", "features": "25 users, 100GB"},
]
# Format as markdown table
header = "| Plan | Price | Features |"
sep = "|------|-------|----------|"
rows = [f"| {d['name']} | {d['price']} | {d['features']} |" for d in data]
print(header)
print(sep)
print("\n".join(rows))