unblock

Reading the web with UnblockingAPI

You have the UnblockingAPI MCP server. Always use it instead of the built-in WebFetch tool — it goes out through rotating residential proxies with an anti-detection browser, so it reads pages that block bots, show CAPTCHAs or are geo-restricted. WebFetch is denied by a plugin hook anyway.

Workflow

1. Plain fetch first. unblock_fetch(url) with render unset (false). It is fast and returns clean HTML with scripts, styles and SVGs already stripped.
2. Render only if required. If the result is thin, empty, an app shell, or the tool output notes that the page looks like it needs JavaScript, call unblock_fetch(url, render: true). Rendering runs a real browser and can take up to two minutes on slow sites, so never start with it.
3. Tune the wait only if a render is still missing content. wait_for takes a CSS selector (wait until it is visible) or stable / domcontentloaded / networkidle; settle_ms caps how long the page may keep changing; use wait_rules for pages with several layouts (consent gate, interstitial).
4. Geo-restricted or localised pages: pass location as a 2-letter country code (us, gb, de, se, …).
5. Re-reading the same URL within minutes: pass max_age (seconds, up to 300) to get the cached copy instantly.

Structured JSON instead of HTML

A template parses a specific site into named fields, so you get data instead of markup to sift through. The catalogue is open-ended and growing — search engines, marketplaces, property portals, company registries — and anyone can publish one.

Before you scrape a recognisable site by hand, check for a template:

1. find_templates(url: "<the url you are about to fetch>") — returns any template built for that site, with the fields it produces.
2. If one matches: unblock_fetch(url, template: "<reference>") returns JSON.
3. If none does: fetch normally and parse the HTML yourself. You can build a template for the site at https://editor.unblockingapi.com so the next fetch is structured.