firecrawl-monitoring

Firecrawl Monitoring

Firecrawl is an external fetch + change-detection backend for scrape sources our own pipeline can't reach: Firecrawl scrapes the page on a schedule (its proxies clear the anti-bot challenge), diffs each check, runs an AI meaningfulness judge, and POSTs us a webhook that becomes a release. Full architecture: docs/architecture/firecrawl-monitoring.md.

It is not a new source type — it's a per-source toggle on an existing scrape source, stored under source.metadata.firecrawl. source.url stays the human-readable page.

Prod-only. The Firecrawl bindings exist only in production (staging shares prod's Secrets Store, so a staging sync would mutate prod monitors — never add the bindings to [env.staging]).

When to use it

• The page is behind a Cloudflare Managed Challenge (or similar) that our Browser Rendering can't clear — render "succeeds" with a challenge shell, no_change / 0 releases on a page that's clearly updating.
• A client-rendered page keeps failing the render path and there's no feed, no .md view, no GitHub source (finding-changelogs exhausted its ladder).

When NOT to use it:

• A feed/GitHub/plain-scrape source that works — Firecrawl costs credits per check; the in-repo pipeline is free.
• First resort on a rendering hiccup. Run the render dry-run first (releases source fetch <source> --dry-run) — a populated candidate count means the normal pipeline can see the page and something else is wrong.
• Staging, or any environment without the prod bindings.

Enabling a monitor

# scrape monitor (default) — watches ONE multi-entry index page
curl -sS -X POST "$RELEASES_API_URL/v1/sources/<src_…>/firecrawl/sync" \
  -H "Authorization: Bearer $RELEASES_API_KEY" -H "Content-Type: application/json" \
  -d '{ "enabled": true }'

# crawl monitor — index that links to a separate page per entry
# (replit /updates, docker-desktop, langfuse, resend, …): same command with
#   -d '{ "enabled": true, "target": "crawl" }'