bda-research/node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

About the project

bda-research/node-crawler is a Node.js web crawler that fetches pages and exposes their contents through a server-side jQuery-like interface using Cheerio. It is used to collect and extract data from websites while managing request pools, retries, rate limits, priorities, character encodings, and proxies. Its catalogue skill teaches coding agents how to operate the crawler and avoid common usage mistakes.

This repository also configures its own agents. See what node-crawler tells them →

Open on GitHub

6.8kStars on the repository

2Mods indexed here, across every type

4mo agoLast push, which is what freshness is scored on

MITLicence, which decides whether bodies are shown

Sort Relevance Recently updated Fewest tokens Security grade Repo stars Stars this week Top rated 1 mods

node-crawler

bda-research/node-crawler

Skill Claude CodeCodex

Node.js web crawler for production-grade, large-scale tasks — NOT for simple one-off requests. Use only when: bulk scraping, batch downloading, multi-page crawling, long-running spiders, or complex multi-step workflows that need connection pooling, rate limiting, proxy rotation, automatic retries, and Cheerio parsing.…

not rated 6.8k +2 4mo ago A SkillSpector: pass 115 tokens original MIT