server-log-crawler-analyst

server-log-crawler-analyst is a skill for Codex from sergekostenchuk/seo-llm-skill-cluster. It costs 90 tokens per session (882 once invoked), scanned A, original, MIT.

A skill for checking whether search and AI crawlers visited a public website, using server logs and public files such as robots.txt, sitemaps, RSS, and llms.txt.

In plain words
What is it for?
Use it to summarize crawler requests by page, status, and user-agent type, check public discovery files, label the evidence and its limits, and identify follow-up monitoring work.
Why use it?
It provides a monitoring baseline when tools such as Search Console, analytics, rank tracking, or citation services are unavailable. It also keeps raw IP addresses and private query data out of public reports.

Skill for Codex

Written for Codex: agents/openai.yaml present. Also seen: mentions Codex.

Needs its repository: it runs a file that does not travel with it, so clone the repository first. The line is python3 <codex-skills-dir>/senior-skill-architect/scripts/lint_production_skill.py ./skills/server-log-crawler-analyst.

Good fit Use it to summarize crawler requests by page, status, and user-agent type, check public discovery files, label the evidence and its limits, and identify follow-up monitoring work.

Compare 6 skills from other repositories ↓
Install

Getting it into your agent

It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.

Clone the repo
git clone --depth 1 https://github.com/sergekostenchuk/seo-llm-skill-cluster
agentmods
npx agentmods add skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst

Made for: Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for server-log-crawler-analyst

README.md
[![agentmods](https://agentmods.dev/badge/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst/github.svg)](https://agentmods.dev/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst)
Your own site
<a href="https://agentmods.dev/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst"><img src="https://agentmods.dev/badge/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for server-log-crawler-analyst

Your own site · 80×15
<a href="https://agentmods.dev/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst"><img src="https://agentmods.dev/badge/skills/sergekostenchuk/seo-llm-skill-cluster/server-log-crawler-analyst.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 90 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 882 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00090 $0.00882
Opus 5 $0.00045 $0.00441
Sonnet 5 $0.00018 $0.00176
Haiku 4.5 $0.00009 $0.00088

Measured 11d ago against content hash 5203a4a08912, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-11, from the pricing page.

Security

Grade A, and why

server-log-crawler-analyst scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.

The scan reads SKILL.md. This mod also ships 1 executable file (scripts/analyze_access_log.py), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

Copies of this mod

1 near-identical copy found in the catalogue:

skills/server-log-crawler-analyst/SKILL.md · 90 lines

How it starts

The opening of the file, as written. The whole thing — 90 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Server Log Crawler Analyst

Use this skill when credential-free monitoring evidence is enough, or when credentialed Search Console, rank tracking, analytics, or assistant citation tools are unavailable.

Read references/data-source-tiers.md before producing a monitoring report.

Owns

  • credential-free monitoring baseline;
  • server access-log summary with IP privacy protection;
  • crawler/user-agent observation reports;
  • public HTTP checks for robots, llms.txt, sitemaps, RSS, and representative URLs;
  • data-source tiering;
  • monitoring limitations and unknowns;
  • backlog handoff for Search Console, rank/SERP, analytics, and LLM citation monitoring.

Does Not Own

  • credentialed Search Console or analytics API work;
  • scraping search engines;
  • bypassing bot protections;
  • live firewall or WAF changes;
  • claims about rankings or assistant citations without direct evidence;
  • storing raw IP logs in public artifacts.

Workflow

  1. Identify available evidence: public URL fetches, access-log snippet, exported summary, or synthetic fixture.
  2. Label the evidence tier and collection time.
  3. If logs are provided, summarize requests by path, status, user-agent class, and redacted IP hash.
  4. Separate known crawler hits from unknown bot-like traffic.
  5. Check public discovery endpoints when requested.
  6. Record limitations: no Search Console, no rank data, no assistant citation data unless explicitly provided.
  7. Produce a monitoring report using assets/crawler-monitor-report.template.md.
  8. Hand off credentialed or citation tasks to the backlog.

Non-Negotiables

  • Do not publish raw IP addresses.
  • Do not store credentials, cookies, API keys, or private analytics exports in skill files.
  • Do not infer ranking, indexing, or citation success from server logs alone.
  • Do not claim a user-agent is verified Google/OpenAI/Claude/Perplexity unless verification method is documented.
  • Do not scrape SERPs or assistants against terms of service.
  • Do not change firewall, WAF, robots, or server config from this skill.

Read the full file on GitHub · 90 lines

Files

What ships with it

8 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 11d ago First seen · 90 lines · 90 tokens per session scan A 5203a4a08912

Subscribe to this mod's changes

server-log-crawler-analyst is a skill published in the GitHub repository sergekostenchuk/seo-llm-skill-cluster (39 stars, last pushed 3mo ago), licensed MIT. It adds 90 tokens to every session and 882 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

technocore-chat

Coordinate with other AI agents over plain HTTP GETs — shared rooms, durable notes, long-polling. No POST, no sockets, no client libraries, no account; a fetch tool is enough, and an MCP server fronts the same surface. Use when you need to leave a message for another agent, wait for one, or persist state across your…

flop-labs/technocore-chat · 79 tokens

dialectical-paper-writing

Analyze, design, write, revise, translate, and review empirical or technical research papers through an evidence-bounded dialectical workflow. Use when Codex must develop a research question, integrate reference papers with a user's ideas, build phenomenon-mechanism-conditions reasoning, design methods or experiments…

lijinyang439-arch/dialectical-paper-writing · 93 tokens

wiki-ingest

Compile a raw source (document, URL content, meeting notes, research output) into a wikikit agent wiki — merge-don't-overwrite, log contradictions, maintain cross-references, then lint. Use whenever the user says "add this to the wiki", "ingest this", or drops new source material for a wiki maintained with…

AvalancheAI-labs/agent-wiki-kit · 77 tokens

agent-pitbook

Search and contribute to Agent Pitbook, a structured debugging pit knowledge base for coding agents. Use when Codex is diagnosing recurring local-development failures, sandbox issues, dependency installation errors, Docker or port problems, MCP/tool quirks, browser automation failures, or when a verified debugging…

laozhangzzz/agent-pitbook · 70 tokens

duckduckgo-search

Free keyless web, news, and image search via ddgs.

NousResearch/hermes-agent · 19 tokens

mcporter

List, auth, and call MCP servers/tools from the terminal.

NousResearch/hermes-agent · 16 tokens