fetch

scout:fetch v1.0 — URL Content Retrieval

Purpose

Retrieve content from a URL and return it in an LLM-friendly format (Markdown / text). Automatically select the appropriate tool based on privacy sensitivity, protecting pages that should not be sent to external services.

Usage

Explicit invocation:

/scout:fetch URL

Delegation from scout:search: When the Execute/Assess steps need to deep-dive into URL content, follow this skill's workflow.

Privacy Classification

Before fetching, classify the URL's privacy level based on domain and context.

Level	Definition	Examples
Public	Pages accessible to anyone	Blogs, news sites, official docs, GitHub public repos
Confidential	Non-public pages fetchable via fetch-page.py (local Playwright)	Internal wikis, admin panels, dashboards, BASIC auth sites
Authenticated	Pages accessible only via browser sessions (cookies). fetch-page.py cannot reach them	Post-OAuth pages (Google, Notion, Slack, etc.), SaaS settings

Classification hints:

• localhost, *.internal, *.local, corporate domains → Confidential
• console.*, admin.*, dashboard.*, paths containing /settings, /admin → Confidential
• Paths containing /api/ → Confidential (except public API docs on docs.* or developer.* subdomains)
• *.notion.so, *.slack.com, *.figma.com, *.atlassian.net, mail.google.com → Authenticated
• Known login wall → Try Confidential (fetch-page.py) first; reclassify as Authenticated on failure
• Unknown domain → Confidential (err on the safe side; classify as Public only when certain)

Retrieval Flow

Receive URL
  ↓
Classify privacy level
  ↓
Public (Jina MCP available = API key configured)
  ↓
Jina MCP (500 RPM)
  ├─ [402] → r.jina.ai (keyless) + notice 「⚠ Jina token limit reached — falling back to free tier (20/min). Top up: https://jina.ai/」
  ├─ [401] → r.jina.ai (keyless) + notice 「⚠ Jina API key rejected — falling back to free tier. Re-run setup: bash tools/setup.sh」
  ├─ [429/5xx/403/408/connection failure] → r.jina.ai (keyless) silently
  └─ [OK] → done
  ※ If 402/401 already occurred in this session, skip MCP and omit notice
  ※ All r.jina.ai fallbacks share the same failure chain:
       r.jina.ai fails → fetch-page.py (if installed) → WebFetch (direct) → report failure

Public (no Jina MCP = no API key)
  ↓
WebFetch via r.jina.ai (keyless, 20 RPM)
  ├─ [429] → fetch-page.py + notice
  │          「⚠ Jina rate limit hit (20/min). Free API key unlocks 500/min: https://jina.ai/?newKey」
  ├─ [other errors] → fetch-page.py silently
  └─ [OK] → done
       └─ [fetch-page.py fails or unavailable] → WebFetch (direct)
            └─ [fails] → report failure

Confidential → fetch-page.py only → report failure (external APIs prohibited)
Authenticated → browser-control.py only → report failure