scraperapi-datapipeline

ScraperAPI DataPipeline

DataPipeline is a managed scraping product. You define a project (what to scrape, how often, where to send results), and ScraperAPI runs it on your schedule without you managing proxies, retries, or infrastructure.

When NOT to use DataPipeline

• One-off scrapes of a known URL list → use the Async API — faster, cheaper, no project setup.
• Exploring a site without known URLs → use the Crawler.
• Need results in real-time within your code → Async API is programmable; DataPipeline is scheduled.
• Free plan, need recurring execution → recurring schedules require a paid plan.

Use DataPipeline when: scraping runs on a fixed schedule, the input list is large (up to 100,000 items), results should flow to a webhook automatically, or you want email notifications on job completion.

Base URL and Auth

Base URL: https://datapipeline.scraperapi.com/api
Auth:     ?api_key=YOUR_KEY  (query parameter on every request)

Project Types

Set projectType in the create request to choose what to scrape:

Type	Input
urls	Raw HTML from any URL
urls_with_js	Same but with JavaScript rendering
google_search	Search queries
google_news	Search queries
google_jobs	Search queries
google_shopping	Search queries
google_maps	Search queries
amazon_product	ASINs
amazon_search	Search queries
amazon_offers	ASINs
walmart_product	Product IDs
walmart_search	Search queries
walmart_category	Category IDs
walmart_reviews	Product IDs
ebay_product	12-digit product IDs
ebay_search	Search queries
redfin_listing_for_sale	Listing URLs
redfin_listing_for_rent	Listing URLs
redfin_listing_search	Search result URLs
redfin_agent_details	Agent profile URLs

Creating a Project

import os, requests

API_KEY = os.environ["SCRAPERAPI_API_KEY"]
BASE    = "https://datapipeline.scraperapi.com/api"

project = requests.post(
    f"{BASE}/projects",
    params={"api_key": API_KEY},
    json={
        "name":               "Weekly Amazon price monitor",
        "projectType":        "amazon_product",
        "schedulingEnabled":  True,
        "scrapingInterval":   "weekly",
        "scheduledAt":        "now",
        "projectInput": {
            "type": "list",
            "list": ["B09V3KXJPB", "B08N5WRWNW"]   # ASINs
        },
        "apiParams": {
            "country_code": "us"
        },
        "webhookOutput": {
            "url":             "https://yourapp.com/pipeline-results",
            "webhookEncoding": "multipart_form_data_encoding"
        },
        "notificationConfig": {
            "notifyOnSuccess": "with_every_run",
            "notifyOnFailure": "with_every_run"
        }
    }
).json()

print(f"Project created: id={project['id']}")