huggingface

Hugging Face (huggingface.co)

The model/dataset/space listing HTML pages are server-rendered behind CloudFront but there is no reason to scrape them — Hugging Face ships a public JSON API that backs the same UI and accepts every URL filter verbatim. One-to-one parameter mapping, no auth, no rate-limit in practice (a soft ratelimit: "pages";q=100;w=300 is on the HTML routes, not /api).

Detection

Signal	Value
CDN	CloudFront (x-cache: ... cloudfront.net)
Framework	SvelteKit (huggingface-moon)
Anti-bot	None on /api/*. HTML pages have a soft page rate limit.
Auth	Not required for public models/datasets/spaces.
robots.txt	Allows /api/ crawling.

Architecture

Every listing URL https://huggingface.co/models?<filters> has a direct JSON twin at https://huggingface.co/api/models?<same filters>. The HTML page is just a SvelteKit shell that calls the same endpoint client-side.

• Endpoint: https://huggingface.co/api/models
• Filters map 1:1: pipeline_tag, library, language, license, other, search, author, filter.
• Sort: sort=trendingScore|downloads|likes|createdAt|lastModified + direction=-1|1.
• Page size: limit (max 100 observed, use 100).
• Pagination: cursor-based via the Link response header (<...cursor=...>; rel="next"). Extract the cursor query param from that URL and pass it to the next request.

Also available:

• /api/datasets — same shape, for /datasets listings.
• /api/spaces — same shape, for /spaces listings.
• /api/models/<repo_id> — full model card metadata for a single repo.

Strategy used

• Phase 0 (curl): HTML returns 200 but the data lives in a hydration blob. Not worth parsing — instead checked /api/models?pipeline_tag=text-generation&sort=trendingScore. Got clean JSON. Done.
• Phase 1–2: skipped.

Stealth config that works

None needed. Plain fetch with a generic User-Agent.

await fetch("https://huggingface.co/api/models?...", {
  headers: { "User-Agent": "Mozilla/5.0 torch-scraper" },
});