ember: Instructions file for Codex

AGENTS.md

This file provides instructions for AI coding agents working on the ember project.

Overview

ember is a lightweight headless browser for AI agents. It gives agents the ability to browse the web, search for information, crawl sites, and interact with pages.

• The Python package lives in emb/
• Tests live in tests/
• The CLI entry point is emb.cli:app

Build and test

pip install -e ".[dev]"    # install in development mode with test deps
pytest tests/              # run the full test suite

All tests must pass before committing.

Code conventions

• Use type hints on all function signatures
• Use from __future__ import annotations at the top of every module
• Prefer clear code and short # comments over docstrings
• Only add a docstring when a public interface truly needs it
• Keep functions small and focused
• Lazy-load heavy dependencies (Lightpanda subprocess, etc.)
• All user-facing errors should be human-readable strings, not raw tracebacks
• Never use raw ANSI escape codes (\033[36m) in CLI output — always use Rich markup ([cyan]) so Rich can render or strip them correctly depending on the terminal
• Never use Rich Table for CLI output — it pads all rows to the widest cell, adding trailing whitespace that looks bad. Use console.print() with manual alignment instead

Comment style

• Keep comments short and simple
• Use # comments only
• Comment intent, edge cases, and security decisions
• Do not write long comment blocks, ASCII diagrams, decorative separators, or noisy prose comments

Product standard

• Write secure code by default. Validate input, keep safe defaults, and avoid risky shortcuts
• Prioritize UX. Make the happy path smooth and make failure paths clear

Architecture

emb/
├── scrape.py           # URL → markdown (trafilatura first, Lightpanda fallback)
├── crawl.py            # BFS website crawler with sitemap support
├── search.py           # Web search via DuckDuckGo (no API key)
├── map.py              # URL discovery via sitemaps + page links
├── interact.py         # Browser interaction via Lightpanda + LLM provider
├── agent.py            # LLM-powered structured extraction
├── cli.py              # Typer CLI + interactive session REPL
├── api.py              # FastAPI REST server
├── mcp.py              # FastMCP server for agent frameworks
├── _browser.py         # Lightpanda auto-download, version pinning, SHA-256 verification
├── _http.py            # Redirect-safe HTTP helpers that revalidate each hop
├── _url_validator.py   # SSRF protection — blocks private/loopback/link-local IPs
└── types.py            # Shared dataclasses (ScrapeResult, SearchResult, etc.)
tests/
├── test_core.py        # Integration tests (scrape, search, crawl, map)
├── test_unit.py        # Unit tests with mocked dependencies
├── test_api.py         # FastAPI endpoint tests via TestClient
└── test_cli.py         # CLI tests via Typer CliRunner