fetch-sitemap

Fetch Sitemap URLs

Extract URLs from an XML sitemap with optional regex filtering.

Arguments

• $0: URL (required, must start with http:// or https://)
  • If the URL ends with .xml, use it directly as the sitemap URL (backward compatible)
  • Otherwise, run the auto-discovery logic below
• $1: an extended regex pattern for filtering (optional)

If $0 is empty, display the usage below and stop:

Usage: /fetch-sitemap <url> [pattern]

Examples:
  /fetch-sitemap https://kotlinlang.org/docs
  /fetch-sitemap https://example.com/sitemap.xml
  /fetch-sitemap https://example.com docs
  /fetch-sitemap https://example.com/sitemap.xml 'skills|hooks'

If $0 does not start with http:// or https://, inform the user that a valid URL is required and stop.

Sitemap Auto-Discovery

When the URL does not end with .xml, automatically discover the sitemap by probing the following locations one at a time, stopping as soon as one produces output (do NOT run probes in parallel):

Probes 1–2 — fetch and extract in a single curl:

1. {url}/sitemap.xml — path-specific (e.g., https://kotlinlang.org/docs/sitemap.xml)
2. {origin}/sitemap.xml — site root (e.g., https://kotlinlang.org/sitemap.xml), where {origin} is the scheme + host of the URL

curl -sfL --compressed --connect-timeout 5 --max-time 10 <probe-url> | grep -oE '<loc>[^<]+</loc>' | sed 's/<loc>//;s/<\/loc>//'

If the output is non-empty, the sitemap is found and the URLs are already extracted — skip the Extraction section entirely and go straight to Output. If empty, try the next probe.

Probe 3 — robots.txt (different format, two-step):

1. {origin}/robots.txt — fetch and parse for Sitemap: lines, use the first match

curl -sfL --compressed --connect-timeout 5 --max-time 10 <origin>/robots.txt

If a Sitemap: line is found, use that URL and proceed to the Extraction section.

If none of the probes succeed, report an error to the user and stop: