k8s-debug-pods

K8s Debug Pods

Diagnose and fix issues with Kurtosis pods on Kubernetes.

Quick triage

# See all kurtosis-related pods across namespaces
kubectl get pods -A | grep kurtosis

# Check for problem pods (not Running)
kubectl get pods -A | grep kurtosis | grep -v Running

# Get events for a specific pod
kubectl describe pod <POD_NAME> -n <NAMESPACE> | tail -30

Common pod states and fixes

Pending — Unschedulable

The pod can't be scheduled because of node taints, resource pressure, or affinity rules.

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check node conditions (DiskPressure, MemoryPressure, etc.)
kubectl get nodes -o custom-columns=NAME:.metadata.name,CONDITIONS:.status.conditions[*].type

Fix: Add tolerations to the kurtosis config at ~/Library/Application Support/kurtosis/kurtosis-config.yml or fix the node condition.

ImagePullBackOff

The image tag doesn't exist on the registry.

# Check which image is failing
kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A5 "Image:"

# Verify image exists on Docker Hub
docker manifest inspect <IMAGE>:<TAG>

Fix: Push the correct image tag, or fix the image reference in the code.

CrashLoopBackOff

The container starts but crashes immediately.

# Check container logs
kubectl logs <POD_NAME> -n <NAMESPACE>
kubectl logs <POD_NAME> -n <NAMESPACE> --previous

Evicted

The node evicted the pod due to resource pressure.

# Check which nodes have pressure
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type

# Clean up evicted pods
kubectl get pods -A | grep Evicted | awk '{print $2 " -n " $1}' | xargs -L1 kubectl delete pod

Kurtosis-specific pod types

Pod pattern	Component	Image source
kurtosis-engine-*	Engine server	engine/server/Dockerfile
kurtosis-api (in kt-* namespaces)	API Container (APIC)	core/server/Dockerfile
kurtosis-logs-collector-*	Fluentbit DaemonSet	Pulled from registry
kurtosis-logs-aggregator-*	Vector deployment	Pulled from registry
remove-dir-pod-*	Fluentbit cleanup pods	busybox
files-artifact-expander (init container)	Files artifacts	core/files_artifacts_expander/Dockerfile