wayback-indirect-asset-recovery

Wayback Indirect Asset Recovery

Problem

When archiving content from defunct services, direct asset URLs (images, videos, avatars) often return 404/503 from Wayback Machine even when the pages referencing them were archived. The assets themselves may not have been crawled, but their URLs exist in archived HTML.

Context / Trigger Conditions

• Direct Wayback URL for asset returns 404 or 503
• You have a profile/page URL but not the asset URL
• Recovering media from defunct services (Vine, Tumblr, defunct startups)
• Wayback has the page but not the embedded assets
• CDN-hosted assets that weren't in Wayback's crawl scope

Solution

Step 1: Fetch the Archived Page

import requests
import re

session = requests.Session()
session.headers['User-Agent'] = 'YourCrawler/0.1 (archival research)'

# Fetch archived profile/page
page_url = f"https://web.archive.org/web/20170110/https://example.com/user/{username}"
resp = session.get(page_url, timeout=20, allow_redirects=True)

Step 2: Extract Asset URLs from HTML

# Try multiple patterns - pages structure varies
patterns = [
    # Open Graph image
    (r'og:image["\s]+content="([^"]+)"', 'og:image'),
    # JSON data in page
    (r'"avatarUrl"\s*:\s*"([^"]+)"', 'JSON avatarUrl'),
    (r'"imageUrl"\s*:\s*"([^"]+)"', 'JSON imageUrl'),
    # CDN URLs
    (r'(https?://[^"\s]+cdn\.[^"\s]+\.(jpg|png|mp4))', 'CDN URL'),
    # S3 URLs
    (r'(https?://[^"\s]+\.s3\.amazonaws\.com/[^"\s]+)', 'S3 URL'),
]

for pattern, name in patterns:
    match = re.search(pattern, html)
    if match:
        asset_url = match.group(1).replace('&', '&')
        break

Step 3: Handle Wayback-Wrapped URLs

URLs extracted from archived pages are often already Wayback URLs:

# If URL is already wrapped, convert im_ to id_ for raw content
if 'web.archive.org/web/' in asset_url:
    # Replace /web/TIMESTAMP/ or /web/TIMESTAMPim_/ with /web/TIMESTAMPid_/
    download_url = re.sub(r'/web/(\d+)(im_)?/', r'/web/\1id_/', asset_url)
else:
    # Wrap raw URL in Wayback
    download_url = f"https://web.archive.org/web/{timestamp}id_/{asset_url}"