log-analyzer

Log Analyzer

Overview

Log viewers filter lines. This skill finds patterns.

Use it when an agent runtime, tool process, gateway, scheduler, or other service has produced enough log output that individual grep hits no longer explain the system behavior. The analyzer parses standard log lines, normalizes repeated messages, groups errors by component and tool, and produces a structured anomaly report that is suitable for debugging sessions or cron digests.

The script is intentionally lightweight and portable: it uses only Python's standard library and works on any text log with timestamp, level, optional component, and message fields. It also has a best-effort fallback for unstructured logs.

Quick Start

cd agent-ops/log-analyzer
python scripts/analyze_logs.py scan --log-file agent.log --since 24h

For per-tool failure rates from a Hermes profile's session DB (structured exit_code signals, NOT regex-over-content):

python scripts/state_failures.py                 # last 7 days, dashboard
python scripts/state_failures.py --days 30 --json
python scripts/state_failures.py --quiet         # cron: silent when healthy

See references/state-failure-monitor.md for provenance (adapted concept from hermes-dojo), the spike evidence for why structured signals matter, and interpretation notes.

To write JSON for later Markdown rendering:

python scripts/analyze_logs.py scan --log-file agent.log --since 24h --output scan.json
python scripts/analyze_logs.py report --scan scan.json --output report.md

For cron-compatible anomaly detection:

python scripts/analyze_logs.py scan --log-file agent.log --since 24h --quiet

--quiet exits 0 and prints nothing when no anomalies are found. If anomalies exist, it prints the JSON report so the scheduler can deliver the digest.

What It Detects

1. Error clusters — the same normalized error message repeated 3+ times within the selected time window. URLs, IP addresses, and numbers are replaced with placeholders before grouping so repeated failures with changing IDs still cluster.
2. Rate limit hits — HTTP 429, rate limit, rate_limit, too many requests, and quota exceeded patterns. The analyzer groups them by provider when it can detect a provider name.
3. Timeout patterns — timeout, timed out, deadline exceeded, and connection timeout. Results are grouped by detected tool name and include example URLs where present.
4. Tool failures — error lines grouped by tool name extracted from patterns such as tool: terminal, tool_call: web_search, tool terminal failed, or lines emitted by a tools component.
5. Session crashes — fatal errors, unhandled exceptions, stack traces, Traceback, Exception, and segfault markers. Multiline stack traces are grouped as one crash entry with nearby context.
6. Component breakdown — error and warning counts by component such as gateway, agent, tools, cron, or unknown.
7. Error timeline — error counts bucketed by hour to reveal spikes and regressions after deploys or scheduled jobs.