wayback-cdx-cloud-ip-workaround

Wayback CDX API: Cloud/Datacenter IP Workaround

Problem

The Wayback Machine CDX API (web.archive.org/cdx/search/cdx) silently refuses to serve data to requests from cloud provider IP ranges (Google Cloud Run, AWS, GCE, etc.). There is no HTTP error code — requests either timeout or return 0 bytes. The same queries work perfectly from residential/local IPs.

Context / Trigger Conditions

• CDX API returns empty (0 bytes) or times out from Cloud Run, Lambda, GCE, etc.
• Same query via curl on local machine returns expected data
• No HTTP error codes (not 429, not 403) — just empty or timeout
• Other external APIs (Common Crawl, Arctic Shift, Arquivo.pt) work fine from the same cloud instance
• Even with VPC connector + static IP configured, CDX still fails
• Increasing timeouts and retries doesn't help

Solution

The Pattern: Local Fetch → Cloud Storage → Cloud Import

Since CDX won't respond to cloud IPs, split the workflow:

1. Fetch locally: Run CDX queries from your local machine, save results to a file
2. Upload to cloud storage: gsutil cp the file to GCS (or S3, etc.)
3. Import on cloud: Have your Cloud Run job read from cloud storage instead of CDX

Implementation

Step 1: Local fetch script

# Run this locally — CDX works from residential IPs
import urllib.request, urllib.parse

CDX = 'https://web.archive.org/cdx/search/cdx'
results = set()
resume_key = None

while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'filter': 'statuscode:200',
        'limit': '10000',
        'showResumeKey': 'true',
    }
    if resume_key:
        params['resumeKey'] = resume_key

    url = f"{CDX}?{urllib.parse.urlencode(params)}"
    req = urllib.request.Request(url, headers={'User-Agent': 'MyBot/0.1'})
    with urllib.request.urlopen(req, timeout=120) as resp:
        raw = resp.read().decode('utf-8', errors='replace')

    parts = raw.rstrip().rsplit('\n\n', 1)
    data_lines = parts[0].strip().split('\n')
    resume_key = parts[1].strip() if len(parts) == 2 else None

    for line in data_lines:
        results.add(process_line(line))  # Extract what you need

    if not resume_key or len(data_lines) < 10000:
        break
    time.sleep(3)

# Save to file
with open('/tmp/results.txt', 'w') as f:
    for item in sorted(results):
        f.write(f'{item}\n')