search-engine

Design a search engine. Topic playbook. The agent loads this canned reference architecture and adapts it to the user's actual scale and constraints, then emits the SDD via the generic template.

Use when the problem is: web search, site/doc search at scale, a crawler, or a large retrieval system. For internal doc search at small scale, say so and simplify (often a managed engine is the whole answer).

Ask once if missing: corpus scope (internal docs vs open web), target QPS, freshness SLO, build vs buy appetite.

Read:

• guides/design-method.md
• guides/writing-style.md
• guides/templates/system-design.md
• guides/pipeline.md

Generate the SDD using the template, filled with the reference below, adapted to user scale.

Save to .claude/runtime/outputs/architect/design/{feature_id}.md. Sensors: sensors/design-structure.md, sensors/design-rigor.md. Evals: evals/design-quality.md.

---

Reference architecture (adapt, do not copy blindly)

One line: a search engine is a distributed factory that turns URLs into rankable documents. It is a chain of decisions: what to discover, fetch, store, index, retrieve, promote. Each stage kills bad cost and keeps useful signal.

Four subsystems + two support planes

• Crawler: discover and fetch pages.
• Processing pipeline: parse, clean, extract, enrich.
• Indexing pipeline: build the indexes.
• Query serving: analyze query, retrieve candidates, rank, assemble result.
• Support: metadata/policy (robots.txt, politeness, canonical, scheduling, dedup) and observability/control (metrics, debugging, reprocess, backfill, allow/blocklists).

Crawler (the heart)

• URL frontier is not one queue. Split into: URL-seen store (Bloom filter + persistent), crawl-state store (status, hash, next-recrawl), scheduler priority queues.
• Multi-queue per host: one pending queue per host + a next_eligible_timestamp; a global heap orders hosts by lowest eligible time and highest priority. Gives fairness + politeness together. A single global queue creates hot domains and looks like a DDoS.
• URL canonicalization before enqueue (lowercase host, drop fragments, default ports, clean tracking params, resolve relative, drop session ids). Skipping this explodes duplicates.
• robots.txt cached per host with TTL; respect allow/disallow and crawl-delay. Timeouts, redirect limits, max download size.
• Politeness is not a fixed sleep. Next request after max(min_delay, k * observed_latency, robots_delay); max concurrency per host; exponential backoff on errors.
• Fetcher stateless: async I/O, connection pool, gzip/brotli, conditional GET (ETag / If-Modified-Since) for cheap recrawl, content-type sniffing, body checksum.
• Crawler traps: infinite calendars, e-commerce facets, param explosions, pagination loops. Guardrails: crawl budget per host, fan-out limit per page, param regex blocklists, template-repeat score, depth limit. Without these, 80% of cost goes to the worst 5% of the web.