markcrawl CLAUDE.md

Project instructions for an AI coding assistant working on MarkCrawl, a web crawler that turns JavaScript-rendered web pages into clean, structured Markdown for language-model and search systems.

In plain words
What is it for?
Understanding MarkCrawl’s purpose and output, and following its project-level guidance while working on the crawler.
Why use it?
They give the assistant the project context and a specific rule for commit messages, reducing mistakes when changing or committing code.

Instructions file

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add instructions/aimlpm/markcrawl/claude-md
Clone the repo
git clone --depth 1 https://github.com/AIMLPM/markcrawl
Per session 86 This file is loaded in full into every session.
When invoked 86 The same file — it is already loaded in full.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00086 $0.00086
Opus 5 $0.00043 $0.00043
Sonnet 5 $0.00017 $0.00017
Haiku 4.5 $0.00009 $0.00009

Measured 2d ago against content hash ac74c0388ff8, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

markcrawl CLAUDE.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

CLAUDE.md · 14 lines

What it actually says

CLAUDE.md

Project-level instructions for AI assistants working on MarkCrawl.

Project overview

MarkCrawl is a web crawler that produces clean Markdown for LLM/RAG pipelines. It handles JavaScript-rendered pages, deduplicates content, and outputs structured Markdown ready for chunking and embedding.

Git commits

  • Do NOT add Co-Authored-By lines to commit messages.
Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 14 lines · 86 tokens per session scan A ac74c0388ff8

Subscribe to this mod's changes

markcrawl CLAUDE.md is an instructions file published in the GitHub repository AIMLPM/markcrawl (3 stars, last pushed 12d ago), licensed MIT. It adds 86 tokens to every session, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.