crawlberg

crawlberg is a skill for Claude Code from xberg-io/crawlberg. It costs 107 tokens per session (2,966 once invoked), scanned A, original, MIT.

A web-crawling tool that fetches pages, follows links, finds site URLs, and converts web content into readable Markdown. It can use a headless browser, meaning a browser that runs without a visible window, when needed.

In plain words
What is it for?
Use it to scrape pages, crawl websites, map sitemaps, run batch jobs, download content, or connect the crawler to another agent through MCP.
Why use it?
It brings scraping, crawling, URL discovery, browser interaction, and an MCP server into one local tool.

Skill for Claude Code

Written for Claude Code: shipped in a Claude Code plugin. Also seen: mentions Claude Code; mentions Codex; mentions OpenCode.

Part of the crawlberg plugin — 7 skills, 1 MCP server shipped together

Good fit Use it to scrape pages, crawl websites, map sitemaps, run batch jobs, download content, or connect the crawler to another agent through MCP.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/xberg-io/crawlberg/crawlberg
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add xberg-io/crawlberg --skill crawlberg
Clone the repo
git clone --depth 1 https://github.com/xberg-io/crawlberg

Made for: Claude Code.

Or install crawlberg, the plugin that ships this one along with the rest of its 7 skills, 1 MCP server.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for crawlberg

README.md
[![agentmods](https://agentmods.dev/badge/skills/xberg-io/crawlberg/crawlberg.svg)](https://agentmods.dev/skills/xberg-io/crawlberg/crawlberg)
Your own site
<a href="https://agentmods.dev/skills/xberg-io/crawlberg/crawlberg"><img src="https://agentmods.dev/badge/skills/xberg-io/crawlberg/crawlberg.svg" alt="Measured on agentmods" height="20"></a>
Per session 107 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 2,966 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector warn 7 Sept 2026
SkillSpector: 16 findings, up to medium

These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →

  • medium MCP Rug Pull · line 40
    npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.
    Fix: Pin the version: npx @scope/[email protected]
  • medium MCP Rug Pull · line 41
    uvx/uv tool run commands without ==version create a rug-pull risk.
    Fix: Pin the version: uvx package-name==1.2.3
  • medium Prompt Injection · line 84
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 88
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 89
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 92
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 94
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 96
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 188
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Prompt Injection · line 198
    Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.
    Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
  • medium Agent Snooping · line 248
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
  • medium Agent Snooping · line 250
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
  • medium Agent Snooping · line 252
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
  • medium Agent Snooping · line 254
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
  • medium Agent Snooping · line 256
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
  • medium Agent Snooping · line 258
    Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.
    Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00107 $0.02966
Opus 5 $0.00053 $0.01483
Sonnet 5 $0.00021 $0.00593
Haiku 4.5 $0.00011 $0.00297

Measured 9d ago against content hash b8cea4a39ba2, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-08, from the pricing page.

Security

Grade A, and why

crawlberg scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugin/.ai-rulez/skills/crawlberg/SKILL.md · 260 lines

How it starts

The opening of the file, as written. The whole thing — 260 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Crawlberg

Crawlberg is a Rust-native web crawler and scraper. It fetches static HTML with reqwest, falls back to headless Chrome when a page needs JS or trips a WAF, and converts every result to clean Markdown via the built-in HTML→Markdown engine.

Use this skill when the user wants to:

  • Scrape a single URL to Markdown plus structured metadata.
  • Crawl a site following links bounded by depth, page count, and concurrency.
  • Enumerate URLs from sitemaps without paying for rendering.
  • Drive a real browser (click, type, scroll) and capture the resulting DOM.
  • Run the same operations from another agent harness via MCP tools.

Installation

The plugin shells out to a crawlberg binary on PATH. Install one of:

brew install xberg-io/tap/crawlberg
# or run without a persistent install (the CLI proxy package self-installs the binary):
npx @xberg-io/crawlberg-cli --help
uvx --from crawlberg-cli crawlberg --help
# or build from source:
cargo install crawlberg-cli --features all

The serve and mcp subcommands are gated behind non-default cargo features (api and mcp). The Homebrew tap is built with all features, so both subcommands work out of the box. A from-source build must pass --features mcp (and --features api for serve), or --features all, to include them.

Verify:

crawlberg --version

Headless fallback needs Chrome/Chromium reachable locally (chromiumoxide launches it on demand). Skip the install if you only plan to use --browser-mode never.

Command map

crawlberg scrape <url>          # single page → JSON or Markdown
crawlberg crawl <url...>        # follow links, BFS, depth-bounded
crawlberg map <url>             # enumerate URLs via sitemaps + link extraction
crawlberg interact <url>        # browser actions: click, type, scroll
crawlberg batch-scrape <url...> # scrape many URLs concurrently
crawlberg batch-crawl <url...>  # crawl many seed URLs concurrently
crawlberg download <url>        # download a document, report file metadata
crawlberg citations <input>     # markdown links → numbered citations (text or @file.md)
crawlberg version               # print the crawlberg version as JSON
crawlberg mcp                   # MCP server (stdio) — auto-registered (`mcp` feature)
crawlberg serve                 # REST API server (`api` feature)

Read the full file on GitHub · 260 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 9d ago First seen · 260 lines · 107 tokens per session scan A b8cea4a39ba2

Subscribe to this mod's changes

crawlberg is a skill published in the GitHub repository xberg-io/crawlberg (169 stars, last pushed 2d ago), licensed MIT. It adds 107 tokens to every session and 2,966 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

tree-sitter-language-pack

Parse and extract code intelligence from 371 programming languages using tree-sitter grammars. Use when writing code that parses source, extracts structure/imports/exports/symbols/docstrings/comments, detects a language, runs syntax diagnostics, or produces syntax-aware chunks for LLMs — in Rust, Python…

xberg-io/tree-sitter-language-pack · 95 tokens

extracting-code-structure

Use when the user wants structured code metadata from a source file — functions, classes, imports, exports, symbols, docstrings, comments, or syntax diagnostics. Covers ts-pack process feature flags, the JSON result shape, and the default feature set.

xberg-io/tree-sitter-language-pack · 58 tokens

chunking-for-llms

Use when the user wants to split source code into chunks for an LLM context window without breaking syntax mid-construct. Covers ts-pack process --chunk-size, why syntax-aware splits beat fixed-byte splits, picking a size, and the chunk JSON shape.

xberg-io/tree-sitter-language-pack · 59 tokens

detecting-languages

Use when the user wants to know which programming language a file or snippet is. Covers implicit detection in ts-pack parse/process, confirming support with ts-pack list/info, and the SDK detection functions for path, extension, and raw content.

xberg-io/tree-sitter-language-pack · 60 tokens

managing-parsers

Use when the user needs to manage the tree-sitter parser cache — prefetch parsers for offline or CI runs, list what is downloaded, inspect a language, find the cache directory, or clean it. Covers ts-pack download, list, info, cache-dir, clean, and init.

xberg-io/tree-sitter-language-pack · 70 tokens

using-the-mcp-server

Use when parsing source, extracting code structure, or detecting a language through the tree-sitter-language-pack MCP server's tools, rather than shelling out to the ts-pack CLI. Covers the tool surface, the auto-installing launcher, and when MCP beats the CLI or SDK.

xberg-io/tree-sitter-language-pack · 62 tokens