distributed-web-crawler: Instructions file for Codex

Distributed Web Crawler: Agents & Architecture

This document provides a comprehensive overview of the Distributed Web Crawler architecture, its running components ("agents"), the project structure, what each package is responsible for, and how to operate the system.

1. Project Structure

The project is structured as a Go workspace containing three main modules: scheduler, worker, and common.

distributed-web-crawler/
├── common/                # Shared utilities and configurations
│   ├── common.go          # Database & Broker connection handlers
│   ├── config.go          # Viper-based configuration loading
│   ├── constants.go       # Shared RabbitMQ queue/exchange names
│   └── models.go          # Shared struct definitions (CrawlMessage, UrlData)
├── scheduler/             # The central coordination agent
│   ├── cmd/scheduler/     # Main entry point for the Scheduler
│   └── internal/
│       ├── bloom/         # Redis-backed Bloom Filter for URL deduplication
│       ├── broker/        # RabbitMQ publishers and result consumers
│       ├── db/            # MongoDB batch insertion & Redis job state
│       ├── handlers/      # HTTP handlers for API endpoints (/crawl)
│       ├── metrics/       # Prometheus metrics definitions
│       └── router/        # Gin HTTP routing setup
├── worker/                # The distributed execution agents
│   ├── cmd/               # Main entry point for the Worker
│   └── internal/
│       ├── broker/        # RabbitMQ consumers for processing URLs
│       ├── crawler/       # Core scraping logic, HTML parsing (goquery)
│       ├── metrics/       # Prometheus metrics definitions
│       └── robots/        # robots.txt validation and compliance
├── compose.yaml           # Docker compose file for orchestrating the stack
├── Dockerfile             # Custom RabbitMQ image with plugins
├── prometheus.yml         # Prometheus scraping configurations
└── go.work                # Go workspace definition