Web Crawler/Spider for NodeJS + server-side jQuery ;-)
bda-research/node-crawler is a Node.js web crawler that fetches pages and exposes their contents through a server-side jQuery-like interface using Cheerio. It is used to collect and extract data from websites while managing request pools, retries, rate limits, priorities, character encodings, and proxies. Its catalogue skill teaches coding agents how to operate the crawler and avoid common usage mistakes.
Latest release v2.1.1 · 16 Jun 2026
These files are bda-research/node-crawler's own configuration. They tell Codex and OpenCode how to work on this repository, so they are not mods to install elsewhere. Copy one as a starting point and replace the parts that are about this project.
AGENTS.md A 408 tok