multi-agent-browser-text-extraction

Multi-Agent Browser Text Extraction

Extract large volumes of original text from JavaScript-heavy Chinese websites (e.g. ddc.shengyen.org 法鼓全集) using parallel delegate_task sub-agents, each with browser toolset.

When to Use

• Target site has JS-heavy tree menu navigation (not static HTML)
• Need to extract 10+ pages of verbatim text from different chapters/books
• Site has bot detection that varies by URL path
• Each extraction requires clicking through tree menus to reach specific chapters
• Content must be 100% verbatim (no paraphrasing)

Prerequisites

• delegate_task tool with browser toolset enabled
• Target site URL (usually a single base URL with query-param navigation)
• Mapping of questions/topics to specific book chapters

Workflow

Phase 1: Discovery (single browser session)

1. Navigate to the base URL (e.g. https://ddc.shengyen.org/)
2. Explore the tree menu structure. Note:
   • Which expandable levels exist (e.g. 輯 > 冊 > 章)
   • Whether chapters are loaded via javascript:void(null) links (they usually are)
   • Whether the URL changes when clicking chapters (it usually doesn't — content loads via AJAX)
3. Test browser_console to extract content:

   document.querySelector('[class*="content"]')?.innerText || document.body.innerText.substring(0, 5000)
   

4. Identify any bot detection early. If the site blocks, try different URL paths.

Phase 2: Batch Spawn (parallel extraction)

Divide the chapters into batches of 5-9 questions each. Each batch spawns a delegate_task with toolsets: ["browser"].

Critical passing of context:

• Provide the EXACT chapter names and book numbers (e.g. 05-02 正信的佛教)
• Provide output format template
• Set max_iterations=40-55 for 6-9 chapters
• Include site navigation instructions in the context

Context template:

HOW TO USE THE SITE:
1. Go to https://ddc.shengyen.org/
2. Click "第五輯 佛教入門類" in left sidebar to expand
3. Click the book (e.g. "05-03 學佛群疑") to expand its chapters
4. Click a chapter name from the expanded sub-menu
5. Use browser_console: document.querySelector('[class*="content"]')?.innerText || ...
6. Copy the EXACT text, verbatim

QUESTIONS TO EXTRACT:
{list of questions with book and chapter mapping}

Output format for EACH question:
## Question: {question}
**Source**: 《書名》聖嚴法師 — Chapter Name
**Original Text**:
> {exact, verbatim, can be long}