wayback-machine-raw-content-id-modifier

Wayback Machine Raw Content with id_ Modifier

Problem

When fetching archived JSON API endpoints from the Wayback Machine, you get HTML-wrapped content with the Wayback toolbar instead of the raw JSON response. This causes JSON parse errors like "Expecting value: line 1 column 1 (char 0)" because the response starts with <!DOCTYPE html> instead of valid JSON.

Context / Trigger Conditions

• Fetching archived API endpoints from web.archive.org/web/{timestamp}/{url}
• JSON parsing fails with "Expecting value: line 1 column 1 (char 0)"
• Response content starts with HTML instead of expected JSON
• Crawling archived REST APIs, JSON feeds, or any non-HTML content from Wayback
• Using Python json.loads(), response.json(), or similar JSON parsing

Solution

Add id_ after the timestamp in the Wayback URL to get raw content:

Default (HTML-wrapped):

https://web.archive.org/web/20170112012313/https://vine.co/api/users/profiles/123

Raw content (add id_):

https://web.archive.org/web/20170112012313id_/https://vine.co/api/users/profiles/123

Code Fix Pattern

# BEFORE (broken - returns HTML)
url = f"https://web.archive.org/web/{timestamp}/https://example.com/api/data"

# AFTER (works - returns raw JSON)
url = f"https://web.archive.org/web/{timestamp}id_/https://example.com/api/data"

Other Wayback Modifiers

• id_ - Raw/identity (no modifications, returns original content)
• if_ - Iframe embed mode
• js_ - JavaScript rewriting mode
• cs_ - CSS rewriting mode
• im_ - Image mode

Verification

1. Test the URL with curl to see actual response:

   # Without id_ - shows HTML with Wayback toolbar
   curl -s "https://web.archive.org/web/20170112012313/https://example.com/api/data" | head -5
   
   # With id_ - shows raw JSON
   curl -s "https://web.archive.org/web/20170112012313id_/https://example.com/api/data" | head -5
   

2. Verify JSON parsing works:

   import json
   import urllib.request
   
   url = f"https://web.archive.org/web/{timestamp}id_/{api_url}"
   with urllib.request.urlopen(url) as resp:
       data = json.loads(resp.read())  # Should work now