searxng-mcp: Instructions file for Codex

AGENTS.md — searxng-mcp

MCP server for private web search via a self-hosted SearXNG instance. Reranks results with a local ML model, fetches full-page content via a four-tier cascade (Firecrawl → Crawl4AI → raw HTTP → Wayback Machine opt-in), and optionally expands queries and synthesizes summaries via Ollama.

What it does

Exposes seven MCP tools:

• search — queries SearXNG, reranks results with a local ML model, returns top N structured results
• search_and_fetch — same as search but also fetches full content of the top result(s) via the fetch cascade
• search_and_summarize — search, fetch, then synthesize a summary with citations via Ollama (default qwen3:14b, override with OLLAMA_SUMMARIZE_MODEL). If the LLM is unavailable it degrades to raw fetched pages, and says so: the payload leads with a --- summarization unavailable (<kind>: <detail>) --- marker. Absence of that marker means you got a real synthesis.
• fetch_url — fetch and extract readable markdown from any public URL; GitHub URLs use the GitHub API, YouTube/Reddit URLs have opt-in fast paths; optional target_selector/wait_for_selector and max_tokens budget
• crawl_site — crawl a site and return a page manifest (Firecrawl → sitemap → optional BFS); content cached for follow-up fetch_url
• clear_cache — purge the Valkey result cache (search, fetch, crawl, or all)
• domain_stats — read-only view of the domain capability database (single-domain or aggregate) with MCP structured output

Structure

src/
  index.ts        # Entry point — creates MCP server, registers tools, selects transport
  http-transport.ts # HTTP transport: bearer gate, /health, per-session routing + eviction
  tools.ts        # Tool definitions (schemas + handlers)
  search.ts       # SearXNG client
  fetch.ts        # Fetch orchestrator: fast paths, robots gate, tier cascade, caching, post-extract
  fetch-utils.ts  # Shared primitives: USER_AGENT, assertPublicUrl, safeFetch, FetchTuning, readBoundedText, TierResult
  ssrf-guard.ts   # isPrivateOrReservedAddress + DNS-validating undici dispatcher (connect-time, redirect-hop safe)
  youtube.ts      # YouTube transcript fast path (timedtext; opt-in via robots)
  reddit.ts       # Reddit .json fast path (post + top comments; opt-in via robots)
  tiers/          # Tier handlers (one file per tier)
    firecrawl.ts  # Tier 1: Firecrawl JS-rendering scrape
    crawl4ai.ts   # Tier 2: Crawl4AI headless fetch + Readability comparison
    raw.ts        # Tier 3: raw HTTP + Readability; fetchRawHtmlForMetadata for post-extract
    wayback.ts    # Tier 4: Wayback Machine CDX lookup + rawFetch (opt-in, WAYBACK_ENABLED)
    github.ts     # GitHub blob/README API fetch
    index.ts      # Barrel re-export
  reranker.ts     # Jina-compatible reranker client + recency weighting
  ollama.ts       # Ollama client (query expansion + summarization)
  cache.ts        # Valkey/Redis caching layer (get/set + Lua EVAL compare-and-set + SCAN)
  domain-db.ts    # Per-domain capability records (tier stats, capabilities); best-effort writes
  domain-stats.ts # Bounded SCAN enumerate + aggregate helpers, summary/formatters (tool + job)
  domain-snapshot.ts # Durable JSON snapshot write/prune/load + non-clobbering restore
  domains.ts      # Domain boost/block filtering + profiles
  config.ts       # Environment variable configuration
  types.ts        # Shared type definitions
  cli/
    dump-domain.ts           # Print one domain's capability record
    domain-db-maintenance.ts # Standalone job: OTel gauges + durable snapshot (cron/PM2, single writer)
    restore-domain-db.ts     # Re-seed the domain-db from the newest snapshot after a flush
tests/
  *.test.ts       # Vitest unit tests (564 tests across 51 files)