agentmods.dev

shantycrawl-mcp AGENTS.md

shantycrawl-mcp AGENTS.md is an instructions file for Codex, OpenCode from schlemperdev/shantycrawl-mcp. It costs 1,820 tokens per session, scanned A, original, MIT.

Repository instructions for ShantyCrawl MCP, a server that connects coding agents to Firecrawl, a web-crawling service, while enabling advanced tools only when needed.

In plain words
What is it for?
Maintaining or extending the Node.js and TypeScript server, its web-scraping tools, dynamic tool activation, and Firecrawl connection.
Why use it?
It defines the server's purpose, communication method, authentication flow, and local implementation rules.

Instructions file for CodexOpenCode

not rated 1 2mo ago A scan 1,820 tokens original MIT
View the source file schlemperdev/shantycrawl-mcp
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add instructions/schlemperdev/shantycrawl-mcp/agents-md
Clone the repo
git clone --depth 1 https://github.com/schlemperdev/shantycrawl-mcp

Made for: Codex, OpenCode.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for shantycrawl-mcp AGENTS.md

README.md
[![agentmods](https://agentmods.dev/badge/instructions/schlemperdev/shantycrawl-mcp/agents-md.svg)](https://agentmods.dev/instructions/schlemperdev/shantycrawl-mcp/agents-md)
Your own site
<a href="https://agentmods.dev/instructions/schlemperdev/shantycrawl-mcp/agents-md"><img src="https://agentmods.dev/badge/instructions/schlemperdev/shantycrawl-mcp/agents-md.svg" alt="Measured on agentmods" height="20"></a>
Per session 1,820 This file is loaded in full into every session.
When invoked 1,820 The same file — it is already loaded in full.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.01820 $0.01820
Opus 5 $0.00910 $0.00910
Sonnet 5 $0.00364 $0.00364
Haiku 4.5 $0.00182 $0.00182

Measured 4d ago against content hash 488fc67771e5, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

shantycrawl-mcp AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

AGENTS.md · 145 lines

How it starts

The opening of the file, as written. The whole thing — 145 lines — stays where its author put it; the contents beside it link to each section on GitHub.

ShantyCrawl MCP -- Dynamic Firecrawl MCP Server

Purpose

Custom MCP server for Firecrawl with lazy-loading tool architecture. Replaces the official firecrawl-mcp (28 tools upfront) with 6 base tools (scrape, crawl, search, check_crawl_status, tool_enable, tool_disable) and dynamic activation of advanced tools via tool_enable/tool_disable. Drastically reduces context consumption.

Ownership

  • Published as shantycrawl-mcp on npm.
  • Source at ~/projects/shantycrawl-mcp/.
  • Stack: Node.js + TypeScript + @modelcontextprotocol/sdk
  • Transport: stdio
  • API target: FIRECRAWL_API_URL env var (default https://api.firecrawl.dev if FIRECRAWL_API_KEY is set, else http://localhost:3002).

Local Contracts

  • Built with @modelcontextprotocol/sdk. No FastMCP or other wrappers.
  • Communication exclusively via stdio.
  • In-memory state tracks which advanced tools are active in the session.
  • tool_enable adds tool to state and emits notifications/tools/list_changed.
  • tool_disable removes tool from state and emits notifications/tools/list_changed.
  • Authentication: FIRECRAWL_API_KEY env var → Authorization: Bearer header on all requests. Never exposed in tool schemas.
  • Extraction tools (scrape, crawl, etc.) call Firecrawl API at the configured FIRECRAWL_API_URL via HTTP fetch, return formatted Markdown.
  • POST/PATCH tools use optional buildBody mapper for argument shaping before API call. No mapper = raw args sent unchanged (fallback-safe).

Work Guidance

  • This server lazy-loads 22 advanced tools via tool_enable. Only 6 base tools appear in tools/list initially. Call tool_enable("<name>") to activate map, extract, parse, agent, interact, research_*, or monitor_* tools.
  • The server also exposes a shantycrawl-setup prompt with full usage instructions — fetch it via GetPromptRequestSchema when connecting.
  • Build output goes to dist/. Config reference lives in ~/shantio/dotfiles/opencode/opencode.jsonc.
  • AGENTS.md is the single source of project contracts — update when purpose, scope, or workflow changes.
  • To add a tool: register schema in tools.ts, add route in firecrawl.ts.
  • Keep schemas minimal (≤4 word descriptions, only vital params).
  • All routes target Firecrawl v2 API. Add buildBody to Route when API body shape differs from user schema.
  • GET tools with query params use URLSearchParams in path function.
  • scripts/coderabbit-loop.ts automates CodeRabbit fix loop: polls PR reviews, applies inline suggestions, commits/pushes until green light or MAX_LOOPS (5). Invoke via tsx scripts/coderabbit-loop.ts <PR_NUMBER> [BRANCH] after git-workflow opens PR. Exit 0 → safe to merge via git-merge. Exit 1/2 → manual.

Read the full file on GitHub · 145 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 4d ago First seen · 145 lines · 1,820 tokens per session scan A 488fc67771e5

Subscribe to this mod's changes

shantycrawl-mcp AGENTS.md is an instructions file published in the GitHub repository schlemperdev/shantycrawl-mcp (1 stars, last pushed 2mo ago), licensed MIT. It adds 1,820 tokens to every session, about $0.0091 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automationclaude-codefirecrawlmcpmodelcontextprotocolopencodeweb-scraping
Related

Other instructions, from other repositories

rayobrowse AGENTS.md

Instructions for rayobyte-data/rayobrowse, covering agents.md, what this project is, prerequisites, local setup and primary usage pattern.

rayobyte-data/rayobrowse · 1,011 tokens

one-search-mcp AGENTS.md

Instructions for yokingma/one-search-mcp, covering agent guide, start here, project map, implementation rules and tests and verification.

yokingma/one-search-mcp · 583 tokens

docpull AGENTS.md

AGENTS.md instructions for raintree-technology/docpull: DocPull is the open-source context dependency manager for AI agents. Its core contract is local-first, browser-free by default, reproducible, cited, and budget guarded.

raintree-technology/docpull · 159 tokens

mcp AGENTS.md

Instructions for fouradata/mcp, covering agents.md, what this server does, tools (four), prompts (six) and errors.

fouradata/mcp · 648 tokens

dompruner-mcp AGENTS.md

AGENTS.md instructions for dong7812/dompruner-mcp, covering agents.md — ai pr review standards, must check (🔴 critical blockers), suggestions (🟡 non-blocking) and out of scope (skip these).

dong7812/dompruner-mcp · 154 tokens

dompruner-mcp CLAUDE.md

Claude Code instructions for dong7812/dompruner-mcp: PR을 생성할 때는 항상 아래 순서를 따른다. 사용자가 "PR 올려줘"라고 하면 별도 지시 없이 merge 완료까지 자율적으로 진행한다.

dong7812/dompruner-mcp · 181 tokens