parallel-subagent-fanout-rate-limit-recover-from-disk

Parallel subagent fan-out: rate limit, and recover from disk not return status

Problem

You dispatch a large fan-out of subagents in one message — each reads a chunk, does work, and writes an output file — to maximize parallelism. Two things go wrong:

1. Above a low concurrency ceiling the API returns "Server is temporarily limiting requests (not your usage limit) · Rate limited" for most of the batch.
2. When you go to retry the "failed" ones, the agents' return status is misleading: many agents that returned a rate-limit/terminal error had already completed their work and written their output file — the error hit on the final summary turn, after the side effect.

Re-running based on the agent return status re-does work that's already on disk (wasted tokens) or, worse, you trust "20 failed" and redo all 20 when 14 actually succeeded.

Context / Trigger Conditions

• Dispatching >~5-6 concurrent general-purpose Agent/Task subagents (or a Workflow fan-out) in a single message.
• Result array peppered with API Error: Server is temporarily limiting requests (not your usage limit) · Rate limited. Some agents show tool_uses: 14-24, subagent_tokens: 0 — they did work but the final return failed.
• Each subagent's contract is to write a file (chunk JSON, transformed doc, etc.) at a known path.

Root cause

1. Concurrency, not quota. Too many simultaneous subagent inference streams trip a server-side throttle that is explicitly "not your usage limit." A smaller batch (≤4-5) stays under it.
2. Side effect precedes the failing turn. A subagent does Read→…→Write (the deliverable) and then emits a final summary turn. The rate-limit/terminal error lands on that last turn, so the file is already on disk even though the agent "errored."

Solution

• Batch at ≤4-5 concurrent for large fan-outs. Dispatch batch, wait, dispatch next. (One validation batch first to prove the prompt, then scale in small batches.)
• To decide retries, CHECK DISK, not the agent return. Enumerate the expected output paths; re-dispatch only the chunks whose files are missing or empty:

  for n in $(seq -w 1 N); do
    f="out/.chunk_${n}.json"
    [ -f "$f" ] && [ -s "$f" ] || echo "MISSING $n"
  done
  

• Make each subagent write to a deterministic absolute path (so this disk check is possible) and validate the file parses before counting it done.
• If the run is paused/resumed across sessions, commit the produced files (or rely on a persistent working dir) so the expensive partial work survives.