ketch CLAUDE.md

Ketch

Fast, stateless CLI for agentic web search and scrape. Single Go binary, no daemon.

Architecture

See AGENTS.md for full module layout and design principles, and design/DESIGN.md for the reasoning behind them — read its Non-Goals & Scope before adding a feature.

• cmd/ — Cobra CLI (root, search, code, docs, scrape, extract, crawl, config, cache, browser, doctor, mcp, version)
• code/ — code.Searcher interface with Grep (built-in default), Sourcegraph, and GitHub backends
• docs/ — docs.Searcher interface with Context7 backend (local FTS5 planned)
• search/ — Searcher interface with Brave (built-in default), DDG, SearXNG, Exa, Firecrawl, Keenable, Tavily, Parallel, and SerpBase backends
• scrape/ — HTTP fetch + browser fallback via Rod for JS-rendered pages
• extract/ — readability + html-to-markdown pipeline, JS shell detection heuristic
• crawl/ — BFS/sitemap crawler with background execution and status tracking
• config/ — JSON config at ~/.config/ketch/config.json
• cache/ — TTL-based page cache backed by bbolt (~/.cache/ketch/cache.db)

Build & Test

make build          # builds ./ketch
make lint           # golangci-lint (gocyclo max 15)
make test           # go test ./...

Pre-commit hook (.githooks/pre-commit) runs gofmt, vet, lint, and tests. Git is configured to use .githooks/ as hooks path.

Output Format

Default output uses YAML frontmatter + markdown (cymbal style):

• ketch scrape — frontmatter (url, title, words) + markdown body
• ketch search — frontmatter (query, backend, result_count) + result list. Under --multi the frontmatter uses backends: (plural, the engines that contributed) plus a failed: key for partial failures, and each result gains a found in: line / backends JSON field
• --json flag available on all commands for structured JSON output

Output Flags (all commands)

Flag	Scope	Default	Description
--max-chars N	scrape, search --scrape	0 (off)	Truncate markdown output to N chars, appends [truncated]
--trim	scrape, search --scrape	false	Strip markdown formatting syntax, keep content text only (~30-40% token reduction)
--minimal	search, code, docs	false	One result per line, tab-separated url/title/snippet, no frontmatter (a 4th backends column is appended under search --multi, plain search only — --scrape --minimal keeps 3 columns)
--multi[=list]	search	—	Federated search across backends (RRF fusion, URL-dedup'd); bare/=all = every usable backend, =brave,exa = a set; mutually exclusive with --backend
--select <css>	scrape	—	Extract only elements matching CSS selector (skips readability)
--no-llms-txt	scrape	false	Disable automatic /llms.txt detection for bare domains
--raw	scrape	false	Output raw HTML instead of markdown (skips /llms.txt; incompatible with --select/--trim)
--force-browser	scrape	false	Always render via the configured browser, skipping JS-shell auto-detection (errors without a browser); composes with --raw and --select
--cookie-file <path>	scrape, search --scrape, crawl	off (config cookie_file)	Netscape cookies.txt jar; matching cookies attached at both fetch layers. Overrides cookie_file; empty value disables. Values never printed