scraperapi-python-sdk

ScraperAPI — Python SDK Best Practices

Requires: Python 3.7+, pip install scraperapi-sdk, SCRAPERAPI_API_KEY environment variable.

Setup

import os
from scraperapi_sdk import ScraperAPIClient

client = ScraperAPIClient(os.environ["SCRAPERAPI_API_KEY"])

Never hardcode the API key. Read it from the environment every time.

Decision Guide

Pick the right call pattern before writing any code.

Situation	Pattern
Single URL, response needed immediately	client.get() — synchronous
20+ URLs, or single URL that often times out	Async Jobs API — requests.post() to async.scraperapi.com/jobs
Supported platform (Amazon, Google, Walmart, eBay, Redfin)	Structured Data endpoint — api.scraperapi.com/structured/<vertical>
Page loads content via JavaScript (React, Vue, Angular)	client.get() with render=True
Site blocks standard proxies	client.get() with premium=True

Use the sync SDK for simple, single-URL work. Switch to raw requests against the async endpoint for batches — the SDK has no built-in async or batch support.

Basic Usage

# Simple scrape — returns HTML as a string
html = client.get(url="https://example.com/")

# With parameters
html = client.get(
    url="https://example.com/",
    params={"render": True, "country_code": "us"}
)

Parameter Reference

Rendering

# Render JavaScript before returning HTML
# Use when: page uses React/Vue/Angular, or initial scrape returns empty/partial content
# Cost: +10 credits (standard: 1 → rendered: 10)
html = client.get(url="https://spa-site.com/", params={"render": True})

# Wait for a specific element before capturing (requires render=True)
# Use when: content loads lazily and simple render still misses it
html = client.get(
    url="https://spa-site.com/",
    params={"render": True, "wait_for_selector": ".product-list"}
)

# Capture a screenshot (auto-enables rendering)
html = client.get(url="https://example.com/", params={"screenshot": True})