tiny-web-crawler

A bounded web-crawling guide for fetching readable page text, searching it, and following relevant links from one or more starting pages. It includes scripts for inspecting one page or crawling several pages up to set limits.

In plain words
What is it for?
Use it to inspect a supplied website, find phrases or facts within pages, follow related links, and collect focused information from a small web crawl.
Why use it?
It provides a repeatable way to gather information from websites while stopping when the answer is found or the crawl limit is reached. This avoids manually fetching unrelated pages.

Skill for Claude CodeCodex

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add skills/leon-ai/leon/tiny-web-crawler
Any agent
npx skills add leon-ai/leon --skill tiny-web-crawler
Clone the repo
git clone --depth 1 https://github.com/leon-ai/leon

Made for: Claude Code, Codex.

Per session 42 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 856 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00042 $0.00856
Opus 5 $0.00021 $0.00428
Sonnet 5 $0.00008 $0.00171
Haiku 4.5 $0.00004 $0.00086

Measured 3d ago against content hash a034f66eb774, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

tiny-web-crawler scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

The scan reads SKILL.md. This mod also ships 2 executable files (scripts/crawl-web.mjs, scripts/fetch-page.mjs), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/agent/tiny-web-crawler/SKILL.md · 95 lines

How it starts

The opening of the file, as written. The whole thing — 95 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Tiny Web Crawler

Use this skill to inspect web pages by fetching content, searching within it, and following relevant links from the starting page.

Scripts

Use the bundled scripts for the actual web fetching and bounded crawling:

  • scripts/fetch-page.mjs: fetch one page, extract compact readable text, links, and query snippets.
  • scripts/crawl-web.mjs: crawl from one or more start URLs, follow relevant links, and stop at limits or strong matches.

Run scripts with node. Leon's shell tool injects runtime shims, so node uses Leon's managed Node.js binary when available and falls back to PATH only when the managed binary is missing.

Example:

node scripts/crawl-web.mjs --url "https://example.com" --query "target phrase" --max-pages 8 --max-depth 2

Do not build $LEON_HOME/bin/node/... or /bin/node/... paths manually.

Workflow

  1. Clarify the target only when the requested information or starting point is ambiguous.
  2. Start from the owner-provided URL when one is given.
  3. Use scripts/crawl-web.mjs to fetch pages, search within content, and follow relevant links.
  4. Use scripts/fetch-page.mjs for one-off page inspection or deeper inspection of a promising page.
  5. Start with compact fetches. Read full or later text chunks only when snippets, title, or links show the page is likely relevant.
  6. Search within fetched content for exact names, phrases, dates, numbers, headings, or nearby synonyms.
  7. Track visited URLs and do not revisit the same page.
  8. Stop as soon as the target information is found with enough context to answer.
  9. If the limit is reached, report what was checked and what remains unresolved.

Limits

Default limits unless the owner specifies otherwise:

  • Max pages: 8
  • Max link depth from the starting page: 2
  • Max pages from the same domain: 5

Prefer stopping early over crawling broadly.

Progressive Fetching

fetch-page.mjs returns compact output by default:

  • textPreview: short readable preview
  • snippets: query matches with nearby context
  • links: normalized URLs with short labels and context
  • chunk.hasMore and chunk.nextOffset: use these to fetch more text only when needed

Read the full file on GitHub · 95 lines

Files

What ships with it

2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 3d ago First seen · 95 lines · 42 tokens per session scan A a034f66eb774

Subscribe to this mod's changes

tiny-web-crawler is a skill published in the GitHub repository leon-ai/leon (17,476 stars, last pushed 7d ago), licensed MIT. It adds 42 tokens to every session and 856 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

playwright-cli

Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.

VoltAgent/voltagent · 52 tokens

snipe

Twitter/X 截流。在 coding/AI agent 工具的高流量推广帖下,自动发现→起草→自我批判→护栏→(按 posture)发布 pikiloom 回复,并记录+度量。内容话术统一来自 promo/pitch.md;可由 promo/orchestrate.md 无人值守驱动。.

xiaotonng/pikiloom · 79 tokens

agent-browser

Браузерная автоматизация для интерактивных веб-задач. Используй, когда нужно открыть сайт, заполнить форму, кликнуть, сделать скриншот, залогиниться, спарсить JS-страницу или протестировать веб-приложение. Триггеры — «зайди на сайт», «заполни форму», «нажми кнопку», «скриншот страницы», «войди в аккаунт», «спарси…

smixs/iva-agent · 114 tokens

webiny-admin-local-storage-catalog

Name: LocalStorage Import: import { LocalStorage } from "webiny/admin/local-storage" Source: @webiny/app/features/localStorage/abstractions.ts.

webiny/webiny-js · 17 tokens

scout-web-access

Strato web-access cross-provider per gli Scout (F-2). 5 componenti coordinati — anti-bot robust scrape, LinkedIn session + search, email IMAP poll, multi-Scout workspace claim, freshness focus. Usato come stack base per ogni sweep: lo Scout sceglie il livello di accesso più leggero che funziona, e sale solo quando…

leopu00/job-hunter-team · 86 tokens

recheck-liveness

Check whether a job posting is STILL OPEN without producing false-opens. Replaces the ad-hoc curl (HTTP 200 = "open") that does NOT see the expiry rendered in JavaScript (Ashby/Workday/Greenhouse) nor the LinkedIn authwall (200 for closed ones too). ALWAYS use it in the recheck; never set isopen by hand off a single…

leopu00/job-hunter-team · 88 tokens