ddgs: Instructions file for Codex

AGENTS.md — Guide for Coding Agents

Project Overview

ddgs (Dux Distributed Global Search) is a Python metasearch library that aggregates results from diverse web search services. It provides a DDGS class with methods like text(), images(), news(), videos(), books(), and extract().

• Python: >=3.10 (tested up to 3.14)
• Dependencies: click, primp, lxml
• Package layout: ddgs/ (source), tests/ (tests), optional extras: api (FastAPI), mcp (stdio)

Build / Lint / Test Commands

# Setup (creates .venv, installs dev deps)
python3 -m venv .venv
.venv/bin/pip install -e .[dev]

# Lint (ruff check + mypy type-check)
.venv/bin/ruff check --fix
.venv/bin/mypy --install-types --non-interactive .

# Format
.venv/bin/ruff format

# Run all tests
.venv/bin/pytest

# Run a single test file
.venv/bin/pytest tests/ddgs_test.py

# Run a single test function
.venv/bin/pytest tests/ddgs_test.py::test_text_search

# Make targets (use .venv/bin/python internally)
make lint
make format
make test

Important: Tests make real HTTP requests and include a 2-second pause between tests (autouse fixture). Tests are slow by nature.

Engine Architecture

Each search engine is a subclass of BaseSearchEngine[T] in ddgs/engines/. Key concepts:

• ENGINES dict in ddgs/engines/__init__.py maps category → {name → engine class}.
• DDGS._search() spawns threads via ThreadPoolExecutor, calling engine.search() in parallel.
• Results are deduplicated and ranked by ResultsAggregator and SimpleFilterRanker.
• BaseSearchEngine handles HTTP requests, HTML parsing (lxml), and XPath-based result extraction.
• Engine instances are cached per DDGS instance in _engines_cache.

Pre-commit Hooks

Pre-commit is managed via prek (not pre-commit). Install with prek install. Hooks run on commit:

1. Standard checks (large files, AST, trailing whitespace, etc.)
2. ruff check --fix
3. ruff format
4. mypy --install-types --non-interactive .