scraperapi-async

ScraperAPI Async Jobs API

The Async API submits scraping jobs in the background and retries them for up to 24 hours to maximize success. Results are retrieved by polling a status URL or received automatically via webhook.

When NOT to use Async

• Single URL, result needed immediately → use the Standard API (api.scraperapi.com) — simpler and returns inline.
• Need to follow links across a site → use the Crawler.
• Need recurring scheduled scraping → use DataPipeline.

Use Async when: scraping 20+ URLs, the target site is slow or flaky, you want webhook delivery, or you need to scrape PDFs/images.

Endpoints

Action	Method	URL
Submit single job	POST	https://async.scraperapi.com/jobs
Submit batch (up to 50k)	POST	https://async.scraperapi.com/batchjobs
Check / retrieve job	GET	https://async.scraperapi.com/jobs/<jobId>
Cancel job	DELETE	https://async.scraperapi.com/jobs/<jobId>

Auth: apiKey in the JSON request body (note: apiKey camelCase, unlike the Standard API's api_key).

Single Job

import os, requests, time

API_KEY = os.environ["SCRAPERAPI_API_KEY"]

# Submit
r = requests.post(
    "https://async.scraperapi.com/jobs",
    json={
        "apiKey": API_KEY,
        "url":    "https://example.com/product/123",
        "apiParams": {
            "render":       True,
            "country_code": "us",
        }
    }
)
job = r.json()
# {"id": "...", "status": "running", "statusUrl": "...", "url": "..."}

# Poll
def poll(status_url, interval=5, max_wait=120):
    deadline = time.time() + max_wait
    while time.time() < deadline:
        data = requests.get(status_url).json()
        if data["status"] == "finished":
            return data["response"]["body"]
        if data["status"] == "failed":
            raise RuntimeError(f"Job failed: {data.get('failReason')}")
        time.sleep(interval)
    raise TimeoutError("Job did not finish in time")

html = poll(job["statusUrl"])