bulk-data-export

Bulk Data Export

Paginate a list endpoint at scale, dedup, and write a clean dataset — safely and with a cost estimate up front.

Setup gate

[ -z "${TIKHUB_API_KEY:-}" ] && echo "Set TIKHUB_API_KEY first (see tikhub-onboarding)."

Step 1 — Estimate cost FIRST (mandatory)

Bulk pulls are the biggest credit spender. Before fetching:

# pages = ceil(target_rows / page_size)
curl -s "https://api.tikhub.io/api/v1/tikhub/user/calculate_price?endpoint=<ENDPOINT_PATH>&request_per_day=<PAGES>" \
  -H "Authorization: Bearer $TIKHUB_API_KEY"
curl -s "https://api.tikhub.io/api/v1/tikhub/user/get_user_daily_usage" -H "Authorization: Bearer $TIKHUB_API_KEY"

State the estimated calls + cost and get the user's go-ahead before running.

Step 2 — Paginate safely

• Use the platform's cursor (max_cursor / pagination_token / continuation_token / cursor / cursor+index) — see tikhub-rest-api and the platform skill.
• Loop until has_more is false or the user's target row count is reached (hard cap).
• Add a short delay / concurrency cap ≤4 (QPS 10/sec). Retry transient 429/5xx with backoff.
• Dedup by stable id (e.g. aweme_id / post id) as you go.

Step 3 — Export

• JSON: write the deduped array to output.json.
• CSV: flatten the fields the user cares about (id, author, text, likes, comments, shares, timestamp, url) into output.csv.
• Report row count, pages fetched, duplicates removed, and the file path.

Verification gate

1. Row count ≈ requested (note if the source ran out early).
2. No duplicate ids in the output.
3. File written and non-empty; CSV header matches columns.

Red flags

• Skipping the cost estimate — never start a bulk pull without one.
• Ignoring the user's row cap / has_more (infinite loop, credit burn).
• Silent truncation — always report how many rows were actually fetched vs. requested.