web-search

Web Search

Use this when a small local LLM needs facts from multiple pages and raw fetch results would overflow context.

For crawly-mcp, prefer the built-in bounded path first:

• call page_search(url, query) first when the task is really "find this within a known page or docs site"
• call search(...) to collect candidate URLs
• call fetch(urls=[...], content_format="text") instead of raw HTML
• keep urls small, usually 1..3 per fetch for local models
• answer in concise prose by default
• switch to schema-first extraction only when the task is large enough that direct prose synthesis would overflow context or when the user explicitly asks for structured output
• if schema extraction is needed, reduce extracted records in batches, not all at once

When To Use

• search + fetch + synthesize across multiple pages
• local or small-context LLMs processing crawled pages
• context overflows after a few fetches
• evidence gathering where you need structured per-source notes

Do not use for single-page questions or when search snippets already answer the question. For a single known page or docs entrypoint, use page_search(...) before broader web search.

Default Workflow

1. Start with page_search(url, query) when you already know the site or docs entrypoint.
2. Otherwise run search(...) and keep only the most relevant URLs.
3. Fetch with content_format="text" and keep batches small.
4. Answer in concise prose when the source set is small and clear.
5. Use per-source structured notes only when the task is broad enough to need batching.
6. If structured notes are needed, reduce in small batches, then merge rollups if needed.
7. Keep any intermediate notes until the answer is verified.

crawly-mcp Guidance

For this repo and its Docker image:

• Prefer page_search(url, query) over search(...) + fetch(...) when you already know the target page or docs site.
• Prefer fetch(..., content_format="text") over post-processing HTML yourself.
• Use content_format="html" only when markup structure is the actual task.
• Treat pages[url] as bounded source text, not as something to dump straight back into the next prompt.
• If page_search(...) returns mode="opensearch" or mode="form", keep results_url because it identifies the landed search page.
• Use tools silently; do not narrate tool calls or internal reasoning to the user.
• Final answers should be prose by default. Include raw JSON only on request.