deepscrape: Instructions file for Codex

DeepScrape Agent Notes

Overview

DeepScrape is a TypeScript/Node.js web scraping service built on Express. It provides API endpoints for single-page scraping, schema-based LLM extraction, batch scraping, URL discovery (/api/map), and multi-page crawling with a Redis-backed BullMQ queue. Playwright is the primary scraper with an HTTP fallback, and content can be returned as HTML, Markdown, or text.

Architecture Summary

Entry point: src/index.ts initializes Express, middleware, routes, and the crawl queue worker.

Scrape flow: /api/scrape in src/api/routes/scraper.ts calls ScraperManager in src/scraper/scraper-manager.ts. Process: Playwright scrape → HTTP fallback if Playwright fails → HTML cleaning → optional Markdown/text transform → optional LLM extraction → cache write.

LLM extraction: src/transformers/llm-extractor.ts uses OpenAIService from src/services/openai.service.ts. Config: OPENAI_API_KEY, OPENAI_MODEL (defaults to gpt-4o).

Crawl flow: /api/crawl in src/api/controllers/crawler.controller.ts uses WebCrawler from src/scraper/crawler.ts. Queueing: jobs are managed via src/services/queue.service.ts using BullMQ + Redis. Streaming discovery: CrawlKickoffService in src/services/crawl-kickoff.service.ts can run streaming URL discovery via URLDiscoveryService and enqueue results as they stream in.

URL discovery: /api/map in src/api/routes/map.routes.ts uses URLDiscoveryService for sitemaps, robots.txt, common paths, search, and optional browser crawling.

Batch scraping: /api/batch/scrape in src/api/routes/batch-scrape.routes.ts uses BatchScrapeService in src/services/batch-scrape.service.ts with Redis persistence and optional webhooks.

Key Data Stores

Redis: crawl metadata, job status, batch metadata, and job results. File cache: CacheService stores scrape results on disk (CACHE_DIRECTORY). Logs: written under logs/ via Winston and morgan.

Important Files

Server bootstrap: src/index.ts Routes: src/api/routes/*.ts Controllers: src/api/controllers/*.ts Scraper: src/scraper/* Transformers: src/transformers/* Services: src/services/* Types: src/types/* Utilities: src/utils/*