seo-log-analysis

Server Log Analysis

Analyzes local server log files for crawl budget breakdown. No MCP or external calls required.

Inputs

• file: Absolute path to server log file (Apache Combined, Apache Common, or Nginx access log). If user provides relative path, resolve with Bash: realpath <path>.

Execution

Step 1: Format Detection

Read the first 10 lines of the log file to detect format:

• Apache Combined: %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i" — 9+ fields, has referer and UA in quotes
• Apache Common: %h %l %u %t "%r" %>s %b — 7 fields, no referer/UA
• Nginx: similar to Apache Combined with slight field order differences
• Check for compressed files (.gz) — if detected, inform user to decompress first

Step 2: Parse Log Lines

Use Bash awk to extract fields. For Apache Combined/Nginx format (9 fields):

awk '{
  ip=$1; method_url=$7; status=$9; ua=$0
  match($0, /"([^"]+)"$/, arr)  # Extract UA from last quoted field
  print ip, $7, $9, arr[1]
}' logfile

For Apache Common (7 fields): ip=$1, request=$7, status=$9, ua="unknown"

Step 3: Classify User-Agents

Group each request into categories:

• Googlebot: Googlebot, Googlebot-Image, Googlebot-News, AdsBot-Google
• Bingbot: bingbot, BingPreview, MicrosoftPreview
• Other search bots: Slurp (Yahoo), DuckDuckBot, Baiduspider, YandexBot, Sogou
• AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, anthropic-ai
• Monitoring tools: Pingdom, UptimeRobot, StatusCake, NewRelic, Datadog
• Real users: everything else (browsers: Mozilla, Chrome, Safari, Firefox, Edge)
• Unknown: no UA or unrecognized

Step 4: Calculate Metrics

Using awk/grep on the log file:

1. Total request count
2. Requests by bot category (count per category, % of total)
3. Requests by HTTP status code (200, 301, 302, 404, 500, etc.)
4. Top 20 crawled URLs by frequency — sort by count descending
5. Top 10 crawled path prefixes (first 2 URL segments, e.g., /blog/, /products/) — aggregate by prefix
6. Requests by hour-of-day (extract hour from timestamp field [DD/Mon/YYYY:HH:MM:SS])