A web-crawling tool that follows links and examines websites to find subdomains, URLs, and JavaScript files.

Crawling a domain, finding subdomains and API endpoints, examining JavaScript references, and mapping a web application.

It reduces the manual work of mapping what a website exposes. It helps create a clearer inventory for security checks or application discovery.

crawl — Skill for Claude Code and Codex by SecurityTalent/bugskill-ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add SecurityTalent/bugskill-ai --skill crawl

Clone the repo

git clone --depth 1 https://github.com/SecurityTalent/bugskill-ai

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/securitytalent/bugskill-ai/crawl/github.svg)](https://agentmods.dev/skills/securitytalent/bugskill-ai/crawl)

Your own site

<a href="https://agentmods.dev/skills/securitytalent/bugskill-ai/crawl"><img src="https://agentmods.dev/badge/skills/securitytalent/bugskill-ai/crawl/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/securitytalent/bugskill-ai/crawl"><img src="https://agentmods.dev/badge/skills/securitytalent/bugskill-ai/crawl.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 76 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 2,510 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00076	$0.02510
Opus 5.5	$0.00030	$0.01004
Sonnet 5.5	$0.00015	$0.00502
Haiku 4.5	$0.00008	$0.00251

Measured 2mo ago against content hash d1eb0d7e6bb4, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

crawl scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

The scan reads SKILL.md. This mod also ships 2 executable files (scripts/crawl.py, scripts/install_tools.sh), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Awesome-Claude-Code-Agent-Skills/01-Reconnaissance-and-OSINT/crawl/SKILL.md · 339 lines

How it starts

The opening of the file, as written. The whole thing — 339 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Web Crawling Skill

Overview

This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.

When to Use This Skill

Use this skill when users request:

Crawling or spidering a URL or domain
Subdomain discovery and enumeration
Endpoint and API discovery
JavaScript file extraction and analysis
Web application reconnaissance or mapping
Bug bounty scoping and enumeration
Attack surface discovery

Trigger keywords: crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance

Quick Start

The primary tool is scripts/crawl.py, which provides a unified interface to both hakrawler and gospider:

# Basic crawl - creates crawl_example.com_TIMESTAMP/
python scripts/crawl.py https://example.com

# Deep crawl with subdomain discovery
python scripts/crawl.py https://example.com --depth 3 --subdomains

# Crawl with JS analysis
python scripts/crawl.py https://example.com --js-analysis

# Use only one tool
python scripts/crawl.py https://example.com --tool hakrawler

# Custom output directory and timeout
python scripts/crawl.py https://example.com --output-dir ./results --timeout 600

Note: The script automatically creates timestamped output directories following the pattern crawl_{DOMAIN}_{timestamp}/ unless a custom output directory is specified.

Tool Installation

If the crawling tools are not installed, use the installation helper:

bash scripts/install_tools.sh

Or install manually:

go install github.com/hakluke/hakrawler@latest
go install github.com/jaeles-project/gospider@latest
export PATH=$PATH:~/go/bin

Verify with: python scripts/crawl.py --install

Read the full file on GitHub · 339 lines

Files

What ships with it

3 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 339 lines · 76 tokens per session scan A d1eb0d7e6bb4

Subscribe to this mod's changes

crawl is a skill published in the GitHub repository SecurityTalent/bugskill-ai (13 stars, last pushed 1mo ago), licensed MIT. It adds 76 tokens to every session and 2,510 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automation Security ai-skill ai-skill-scanner ai-skill-security hackerone-reports public-bug-reports vulnerability-reports

crawl next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
crawlSecurityTalent/bugskill-ai	Skill	13	1mo ago	MIT	A	76	Claude Code, Codex
playwright-best-practicescurrents-dev/playwright-best-practices-skill	Skill	418	3mo ago	MIT	A	214	Claude Code, Codex
fullpage-screenshotmad-coders/fullpage-screenshot-ai-skill	Skill	2	2mo ago	MIT	A	69	Claude Code, Codex
legado-book-source-generatorNarylr350/book-source-creator-skill	Skill	202	2mo ago	none	A	84	Codex
serpbase-skillserpbase-dev/serpbase-skill	Skill	3	5mo ago	MIT	A	98	Codex
reddit-skillsbeggarticksarthurtatum121/reddit-skills	Skill	4	23d ago	MIT	A	51	Claude Code, Codex
hasdataHasData/agent-skills	Skill	14	5mo ago	none	A	154	Claude Code, Codex