clear-kv-cache-tiers-in-llm-d-deployment

Reset vLLM Cache in llm-d

Purpose

Clear the KV / prefix cache on all vLLM pods in an llm-d deployment so the next run starts from a clean state. Prefers non-restart methods; falls back to pod restart when necessary.

---

Step 1: Ask for Namespace and Deployment

1. Detect the current namespace and related options:

CURRENT_NS=$(oc project -q 2>/dev/null || kubectl config view --minify -o jsonpath='{..namespace}' 2>/dev/null || echo "")
SUGGESTED_NS=$(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}' 2>/dev/null \
  | tr ' ' '\n' | grep -iE "llm-d|inference|serving|benchmark" | grep -v "^$CURRENT_NS$" | head -5)

Ask the user:

"Which namespace? 1. <CURRENT_NS> (current context) 2–N. <SUGGESTED_NS> (or type any namespace)"

Set NAMESPACE to the user's answer. If the user just confirms without specifying, use CURRENT_NS — the namespace they are currently working in.

1. List llm-d deployments (the ROLE column reveals a disaggregated setup):

kubectl get deployments -n $NAMESPACE -l app.kubernetes.io/part-of=llm-d \
  -o custom-columns="NAME:.metadata.name,ROLE:.spec.template.metadata.labels.llm-d\.ai/role,READY:.status.readyReplicas,DESIRED:.spec.replicas"
# If empty, broaden:
kubectl get deployments -n $NAMESPACE | grep -iE "llm-d|vllm"

• One result → use automatically, inform the user.
• Multiple → ask: "Which deployment? (or 'all' to reset every vLLM pod in the namespace)"
• Disaggregated prefill/decode — you see both a …-prefill and a …-decode deployment (or ROLE=prefill / ROLE=decode): both must be reset. Prefill pods hold KV cache too, so clearing only decode leaves stale prefill KV. Reset every role — the all selector below covers both in one pass, or run Steps 2–4 once per deployment.

Set DEPLOYMENT_NAME to the chosen deployment (used by the specific-deployment selector and the restart/patch steps). Skip it when the user picked all. For P/D, also capture both role names — used in Step 4:

DEPLOYMENT_NAME=<chosen deployment>
# P/D only — read the two role deployment names from the Step 1 listing:
PREFILL_DEPLOYMENT=$(kubectl get deployments -n $NAMESPACE -l llm-d.ai/role=prefill -o jsonpath='{.items[0].metadata.name}')
DECODE_DEPLOYMENT=$(kubectl get deployments -n $NAMESPACE -l llm-d.ai/role=decode -o jsonpath='{.items[0].metadata.name}')