seeker

Seeker — Find Bugs from Production

Pull logs from pods, analyze, raise issues on GitHub.

Process

1. Pull logs from all pods in the namespace:

   kubectl logs --all-containers --since=1h -l app=<app> --tail=500
   

   If kubectl not available, check for log files in common locations:

   • /var/log/, ~/.pm2/logs/, docker logs
   • Cloud: gcloud logging read, aws logs, fly logs

2. Analyze each log stream for:

   • Unhandled exceptions / stack traces
   • Error-level log lines (ERROR, FATAL, CRITICAL, panic, segfault)
   • OOM kills, restart loops, crash backoffs
   • Slow queries (>1s), timeout errors
   • 5xx HTTP responses, connection refused
   • Auth failures, rate limit hits
   • Memory/CPU warnings

3. Deduplicate — group similar errors by stack trace signature. Don't create 10 issues for the same NullPointerException.

4. Raise GitHub issues for each unique bug:

   gh issue create \
     --title "[seeker] <error type>: <brief description>" \
     --body "<structured body>" \
     --label "bug,seeker"
   

   Issue body format:

   ## Source
   Pod: <pod-name> | Container: <container> | Time: <timestamp>
   
   ## Error
   <exact error message / stack trace>
   
   ## Frequency
   <N occurrences in last hour>
   
   ## Impact
   <what's affected — users, API, jobs>
   
   ## Suggested Fix
   <initial diagnosis + suggested approach>
   

5. Verify old fixes — check if previously raised issues are now fixed in production:

   gh issue list --label seeker --state open
   

   For each open issue:

   • Search current logs for the same error signature
   • If error NO LONGER appears in logs → the fix was deployed and worked:

     gh issue close <number> --comment "✅ Verified fixed in production — error no longer appears in pod logs (checked $(date -u +%Y-%m-%dT%H:%M:%SZ))"
     

   • If error STILL appears → leave open, add comment with latest occurrence

6. Report summary: N logs scanned, M unique errors found, K issues created, J issues auto-closed (verified fixed).