Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/xberg-io/crawlberg/crawl-engineergit clone --depth 1 https://github.com/xberg-io/crawlbergWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/xberg-io/crawlberg/crawl-engineer)<a href="https://agentmods.dev/agents/xberg-io/crawlberg/crawl-engineer"><img src="https://agentmods.dev/badge/agents/xberg-io/crawlberg/crawl-engineer.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00008 | $0.00505 |
| Opus 5 | $0.00004 | $0.00253 |
| Sonnet 5 | $0.00002 | $0.00101 |
| Haiku 4.5 | $0.00001 | $0.00051 |
Grade A, and why
crawl-engineer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
What it actually says
When working on the crawling system:
- The crawl engine lives in
crates/crawlberg/src/engine/withcrawl_loop.rs(main crawl loop),builder.rs(engine configuration), andbatch.rs(batch crawling). The engine orchestrates URL frontier management, HTTP fetching, and content extraction. - HTTP client and request handling is in
crates/crawlberg/src/http.rs. Respect robots.txt rules viarobots.rsand sitemap discovery viasitemap.rs. - Rate limiting, caching, user-agent rotation, and tracing middleware live in
crates/crawlberg/src/tower/as Tower service layers (rate_limit.rs,cache.rs,ua_rotation.rs,tracing_layer.rs). - HTML processing pipeline: parsing via
tlcrate, pruning (pruning.rs), normalization (normalize.rs), and markdown conversion (markdown.rs) usinghtml-to-markdown-rs. - Browser-based crawling (feature-gated behind
browser) uses chromiumoxide inbrowser.rs,browser_pool.rs,browser_profile.rs, andbrowser_detect.rs. Interactive page actions live ininteract/. - AI-powered research agent (feature-gated behind
ai) inresearch/withagent.rs,planner.rs, andsynthesizer.rsusingliter-llmfor LLM calls andminijinjafor prompt templates. - MCP server integration (feature-gated behind
mcp) inmcp/exposes crawling as tool calls. - REST API (feature-gated behind
api) inapi/provides HTTP endpoints via axum. - WARC archive output (feature-gated behind
warc) inwarc.rs. - Configuration types in
types/config.rs, crawl results intypes/results.rs, URL discovery intypes/discovery.rs, page metadata intypes/metadata.rs. - Bindings: Python (
crawlberg-py), Node.js (crawlberg-node), PHP (crawlberg-php), WASM (crawlberg-wasm), C FFI (crawlberg-ffi), with shared binding utilities incrawlberg-bindings-common. - CLI entry point in
crawlberg-cli. - All crawling logic stays in Rust core. Bindings are thin wrappers only.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 22 lines · 8 tokens per session scan A 06ca3514ac17
crawl-engineer is an agent published in the GitHub repository xberg-io/crawlberg (168 stars, last pushed today), licensed MIT. It adds 8 tokens to every session and 505 once invoked, about $0.0000 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
e2e-test-writer
E2E spec author — explores a live app via Playwright MCP (browsersnapshot → interact → assert), authors durable spec files, and mandates browser teardown with per-step deadlines. Invoke for the authoring phase of /swe-workbench:test --mode e2e; the verifier runs the specs after.
plinth-architect
Java architecture specialist. Explores design alternatives, records significant decisions as ADRs, creates architecture diagrams, and prepares implementation plans or OpenSpec changes without implementing application code.
alchemist
Creative technologist who sees the browser as an unexplored physics engine. Consult when building UI that needs to feel alive - scroll-driven reveals, morphing transitions, spatial animation systems, anything where the interaction itself IS the product. Thinks in weight, tension, and breath before thinking in code.…
praman-sap-planner-cli
SAP UI5 test planner via Playwright CLI. Token-efficient alternative to MCP planner. Generates test plan + gold-standard spec using CLI commands.
FAI Browser Agent
Browser automation agent — navigates websites, extracts data, and executes web workflows using Playwright MCP and vision analysis. Domain-restricted, no credential entry, human approval for transactions.
Agent Issues
The Agent will analyze and fix the issues in the repository.