How it behaves
Authenticated on a single token, one request at a time, with Retry-After and the rate-limit headers respected rather than retried through. Never in parallel on the same token, and never across multiple accounts to get around a limit.
Discovery uses the git trees API — one request returns every path in a repository — rather than code search, which would cost ten requests for the same answer. File bodies come from raw.githubusercontent.com, which does not spend API quota. The user agent is agentmods-crawler and carries a contact address.
What it keeps
Repository metadata (stars, forks, last push, licence, archived state), the path and name of each mod, its frontmatter, a content hash, and the measurements computed from it. The body itself is stored only where the licence permits reproducing it.
Opting out
Ask, and the repository is removed and excluded from future crawls — the crawl and the discovery both check the block list, so it does not come back on the next run. No justification needed. The listing exists to help people find your work; if you would rather it did not, that settles it.
Submitting
The opposite direction exists too: point the crawler at a repository and it joins the nightly run. It is read, described, measured and ranked exactly like a repository the crawler found itself.
How it runs
The crawler is a service that schedules itself: a daily delta run at 02:00 (new files, changed hashes, star snapshots), a weekly refresh on Sunday (every repository's stars and status, tier recomputed), and a full discovery on the first Saturday of the month. Every run goes through one queue on one token, so nothing ever crawls in parallel.