Borrowing it
Nothing to install: this file belongs to wyattowalsh/mcp-crawl4ai. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/wyattowalsh/mcp-crawl4ai/main/AGENTS.mdgit clone --depth 1 https://github.com/wyattowalsh/mcp-crawl4aiWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/instructions/wyattowalsh/mcp-crawl4ai/agents-md)<a href="https://agentmods.dev/instructions/wyattowalsh/mcp-crawl4ai/agents-md"><img src="https://agentmods.dev/badge/instructions/wyattowalsh/mcp-crawl4ai/agents-md/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/instructions/wyattowalsh/mcp-crawl4ai/agents-md"><img src="https://agentmods.dev/badge/instructions/wyattowalsh/mcp-crawl4ai/agents-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.01254 | $0.01254 |
| Opus 5 | $0.00627 | $0.00627 |
| Sonnet 5 | $0.00251 | $0.00251 |
| Haiku 4.5 | $0.00125 | $0.00125 |
Grade A, and why
mcp-crawl4ai AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 129 lines — stays where its author put it; the contents beside it link to each section on GitHub.
AGENTS.md -- mcp-crawl4ai
AI agent instructions for this project.
Stack
- Runtime: Python 3.13+
- Server framework: FastMCP v3 (
fastmcp>=3.0.0) - Crawler: crawl4ai (
crawl4ai>=0.8.0) --AsyncWebCrawlerwith Playwright/Chromium - Package manager:
uv(always useuv run/uv add/uv sync) - Test runner: pytest with pytest-asyncio (
asyncio_mode = "auto") - Linter: ruff
- Type checker: ty
Key Files
| File | Purpose |
|---|---|
mcp_crawl4ai/server.py |
All server logic -- 11 tools, 2 resources, 3 prompts, lifespan, auto-setup |
mcp_crawl4ai/__init__.py |
Package version via importlib.metadata |
pyproject.toml |
Project config, dependencies, tool config |
tests/conftest.py |
Test fixtures (mock crawler, client) |
tests/test_server.py |
100+ tests -- all in-memory, no browser |
tests/manual/ |
Manual live test scripts (require browser) |
mcp_crawl4ai/py.typed |
PEP 561 typed package marker |
.github/workflows/ci.yml |
GitHub Actions CI (test + lint + typecheck + Codecov) |
.github/workflows/release.yml |
Release workflow (build + PyPI publish + GitHub Release) |
.github/dependabot.yml |
Dependabot for pip + GitHub Actions |
.github/assets/img/ |
Logo, icon, favicon images |
Dockerfile |
Container build for HTTP transport deployment |
.pre-commit-config.yaml |
Pre-commit hooks (ruff + pre-commit-hooks) |
Commands
# Install
uv sync
# Install browser (first time -- or use mcp-crawl4ai --setup)
mcp-crawl4ai --setup
# Run (stdio -- default for Claude Desktop / MCP clients)
mcp-crawl4ai
# Run (HTTP -- defaults to 127.0.0.1; set --host explicitly for external exposure)
mcp-crawl4ai --transport http --port 8000
# Tests
uv run pytest
# Tests with coverage
uv run pytest --cov=mcp_crawl4ai
# Lint
uv run ruff check mcp_crawl4ai/
# Type check
uv run ty check mcp_crawl4ai/
CI / Pre-commit
# Install pre-commit hooks
uv run pre-commit install
# Run pre-commit manually
uv run pre-commit run --all-files
# Build Docker image
docker build -t mcp-crawl4ai .
# Run via Docker (HTTP transport)
docker run -p 8000:8000 mcp-crawl4ai
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 129 lines · 1,254 tokens per session scan A 714c49e37639
mcp-crawl4ai AGENTS.md is an instructions file published in the GitHub repository wyattowalsh/mcp-crawl4ai (0 stars, last pushed 4mo ago), licensed MIT. It adds 1,254 tokens to every session, about $0.0063 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other instructions, from other repositories
next.js AGENTS.md
AGENTS.md instructions for vercel/next.js, covering next.js development guide, codebase structure, monorepo overview, core package: packages/next and other important packages.
codex AGENTS.md
AGENTS.md instructions for openai/codex, covering rust/codex-rs, the codex-core crate, code review rules, crate api surface and model visible context.
vscode buildNext.instructions.md
Working notes and architecture documentation for the new esbuild-based build system in build/next. Use when making changes to the new build pipeline (transpile/bundle commands, NLS plugin, source-map handling, resource copying, or self-hosting watch tasks).
vscode oss-third-party-notices.instructions.md
Instructions for microsoft/vscode, covering vs code oss third-party-notices pipeline, architecture, pipeline flow in ci, applying the notice (cutover) and fallback chain (never fail the build).
langchain AGENTS.md
AGENTS.md instructions for langchain-ai/langchain, covering global development guidelines for the langchain monorepo, corridor security analysis, project architecture and context, monorepo structure and development tools & commands.
deepseek-harness AGENTS.md
AGENTS.md instructions for deepseek-ai/deepseek-harness, covering agents.md, pre-stable apis and released session data, repository layout, commands and host sandbox failures.