web-extract

Web Extract

Turn selected webpages into structured, traceable evidence. Preserve the source URL on every extracted item.

Workflow

1. Open the exact selected URL. Record redirects and use the final canonical URL as source_url when known.
2. Inspect page title, publisher, publication or update date, and the content area relevant to the question.
3. Prefer direct page reading or structured extraction for static pages.
4. Use Playwright or an equivalent browser tool only when content requires JavaScript rendering, interaction, pagination, expansion, or scrolling.
5. Extract only relevant正文 and preserve headings that give it meaning.
6. Extract tables as structured rows and columns. Preserve headers, units, notes, and table captions.
7. Separate page facts from interpretation. Keep exact short excerpts minimal and paraphrase the rest.
8. Record extraction gaps such as login walls, robots restrictions, missing dates, truncated tables, or inaccessible content.
9. Define acceptance before extraction. For dynamic pages, use required_texts, min_content_chars, min_tables, or min_structured_items with fail_on_quality=true.
10. Use table_contains, max_tables, and max_table_rows to keep large comparison pages focused and model-consumable.

Output Contract

{
  "source_url": "https://example.com/final-page",
  "title": "page title",
  "publisher": "source owner or null",
  "published_at": "known date or null",
  "retrieved_at": "retrieval timestamp",
  "content": [
    {
      "heading": "section heading",
      "text": "relevant extracted or faithfully paraphrased content",
      "source_url": "https://example.com/final-page"
    }
  ],
  "tables": [
    {
      "caption": "table caption or null",
      "columns": ["column A", "column B"],
      "rows": [["value A", "value B"]],
      "source_url": "https://example.com/final-page"
    }
  ],
  "extraction_notes": []
}

Dynamic Pages

• Use Playwright as a low-level extraction tool, not as the research methodology.
• Perform the minimum interaction required to reveal the relevant content.
• Prefer wait_for_selector or wait_for_text over fixed sleeps. Use click_texts and auto_scroll only when required.
• Request screenshot_path when visual evidence is part of acceptance.
• Do not submit forms, log in, accept permissions, or trigger external side effects unless explicitly authorized.
• Record the interaction needed to obtain the content.