wayback-cdx-wildcard-pagination

Wayback CDX API: Wildcard Query Pagination

Problem

The Wayback Machine CDX API's page=N pagination parameter returns empty results when combined with wildcard URL queries (url=domain.com/path/*), even though the same query without page= returns data. This causes scripts to incorrectly report "no results found" when there are actually hundreds of thousands of results.

Context / Trigger Conditions

• CDX query with url=*.example.com/* or url=example.com/path/* and page=0 returns 0 bytes
• Same query without page= parameter returns expected data
• Using web.archive.org/cdx/search/cdx API endpoint
• matchType=prefix + collapse=urlkey + page=N combination also fails silently
• Script works with limit=5 but fails when adding page=0

Solution

Option 1: Use showResumeKey=true (Recommended)

The most reliable pagination method. CDX appends a base64 resume token after a blank line separator in the response.

resume_key = None
while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'filter': 'statuscode:200',
        'limit': '10000',
        'showResumeKey': 'true',
    }
    if resume_key:
        params['resumeKey'] = resume_key

    data = fetch_cdx(params)
    if not data:
        break

    # Resume key is after the last blank line
    parts = data.rstrip().rsplit('\n\n', 1)
    if len(parts) == 2:
        data_lines = parts[0].strip().split('\n')
        resume_key = parts[1].strip()
    else:
        data_lines = parts[0].strip().split('\n')
        resume_key = None

    # Process data_lines...

    if not resume_key or len(data_lines) < limit:
        break

Option 2: Use offset=N

Works but less efficient for very large result sets.

offset = 0
limit = 10000
while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'limit': str(limit),
        'offset': str(offset),
    }
    data = fetch_cdx(params)
    lines = data.strip().split('\n')
    if not lines or not lines[0]:
        break
    offset += len(lines)