debug-buttercup

Debug Buttercup

When to Use

• Pods in the crs namespace are in CrashLoopBackOff, OOMKilled, or restarting
• Multiple services restart simultaneously (cascade failure)
• Redis is unresponsive or showing AOF warnings
• Queues are growing but tasks are not progressing
• Nodes show DiskPressure, MemoryPressure, or PID pressure
• Build-bot cannot reach the Docker daemon (DinD failures)
• Scheduler is stuck and not advancing task state
• Health check probes are failing unexpectedly
• Deployed Helm values don't match actual pod configuration

When NOT to Use

• Deploying or upgrading Buttercup (use Helm and deployment guides)
• Debugging issues outside the crs Kubernetes namespace
• Performance tuning that doesn't involve a failure symptom

Namespace and Services

All pods run in namespace crs. Key services:

Layer	Services
Infra	redis, dind, litellm, registry-cache
Orchestration	scheduler, task-server, task-downloader, scratch-cleaner
Fuzzing	build-bot, fuzzer-bot, coverage-bot, tracer-bot, merger-bot
Analysis	patcher, seed-gen, program-model, pov-reproducer
Interface	competition-api, ui

Triage Workflow

Always start with triage. Run these three commands first:

# 1. Pod status - look for restarts, CrashLoopBackOff, OOMKilled
kubectl get pods -n crs -o wide

# 2. Events - the timeline of what went wrong
kubectl get events -n crs --sort-by='.lastTimestamp'

# 3. Warnings only - filter the noise
kubectl get events -n crs --field-selector type=Warning --sort-by='.lastTimestamp'

Then narrow down:

# Why did a specific pod restart? Check Last State Reason (OOMKilled, Error, Completed)
kubectl describe pod -n crs <pod-name> | grep -A8 'Last State:'

# Check actual resource limits vs intended
kubectl get pod -n crs <pod-name> -o jsonpath='{.spec.containers[0].resources}'

# Crashed container's logs (--previous = the container that died)
kubectl logs -n crs <pod-name> --previous --tail=200

# Current logs
kubectl logs -n crs <pod-name> --tail=200