batch-download

Batch Download

Orchestrate multi-agent batch discovery and retrieval: plan the task decomposition, delegate atomic subtasks to parallel sub-agents, gather and verify evidence, then integrate and validate all results into a structured deliverable. The Orchestrator coordinates — sub-agents execute.

Workflow Decision Tree

Determine the entry point based on user input:

User Query
  │
  ├─ Type A — Targets unknown (must search first)
  │   → Phase 1 (Plan) → Phase 2 (Search & Discovery)
  │     → Phase 3 (Parallel Download) → Phase 4 (Integrate & Validate)
  │
  ├─ Type B — Targets already provided (URLs / file list given)
  │   → Phase 1 (Plan) → Phase 2 (URL Validation & Pattern Discovery)
  │     → Phase 3 (Parallel Download) → Phase 4 (Integrate & Validate)
  │
  └─ Hybrid — Some targets known, some must be discovered
      → Phase 1 (Plan) → Phase 2 (Partial Search + Validation)
        → Phase 3 (Parallel Download) → Phase 4 (Integrate & Validate)

All types follow the same 1→2→3→4 sequence. The difference is Phase 2's scope: Type A performs full search, Type B validates known URLs and discovers patterns, Hybrid does both.

Phase 1: Task Decomposition (Orchestrator Only)

Goal: Break the user request into clear, atomic, verifiable subtasks before any execution.

Process:

1. Restate the user's request in operational terms
2. Identify the task type (Type A / Type B / Hybrid) based on whether target URLs are known
3. Decompose into atomic subtasks:
   • Each subtask has one well-defined goal (e.g., "download report for 2021 from URL X")
   • Each subtask is independent where possible (to enable parallelization)
   • Each subtask has explicit success criteria (file type, size, content validation)
4. For multi-dimensional requests (different years / regions / categories / file types / data sources), create one subtask per dimension
5. Plan search subtasks explicitly (if Type A or Hybrid):
   • Do NOT assign "all search" to a single agent when multiple independent searches are possible
   • Parallelize partitionable search spaces: by year, by site, by category, by alphabet range, by region — each partition gets its own search sub-agent
   • Plan retry strategies: if a search attempt fails, plan alternative queries, sites, or filters as new subtasks rather than giving up
6. If the search task is complex, split into multiple sequential stages (e.g., find index page → extract file links → validate → download)