configure-wva-autoscaling-llm-d

Agent Behavior Rules

1. Follow steps IN ORDER. Never skip or combine steps.
2. STOP after each step and ask for explicit permission to proceed to the next step.
3. Do NOT modify existing repository code. Cloning a missing repo is allowed. Exception: the kustomize symlink fix in Step 4b is a known bug fix — apply it if needed.
4. Use existing skill scripts when possible — see scripts/SCRIPTS.md.
5. Before creating any Kubernetes resource, state what will be created and why.
6. After each kubectl/make command, run a verification check and report the result before continuing.

---

Step 1 — Select Target Namespace and Deployments

Ask the user:

"Which Kubernetes namespace should WVA monitor?" (Provide a single namespace, e.g., my-llm-ns)

Export the answer:

export WVA_NS=<namespace>

WVA will be deployed into this namespace so it can watch the llm-d workloads there.

Then discover ALL llm-d decode deployments in that namespace:

kubectl get deployment -n $WVA_NS -l llm-d.ai/role=decode -o custom-columns=NAME:.metadata.name,MODEL:.metadata.labels.llm-d\.ai/model-id,REPLICAS:.spec.replicas

If no results, try the alternative label:

kubectl get deployment -n $WVA_NS -l app.kubernetes.io/part-of=llm-d -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas

Also discover EPPs (InferencePool / EndpointPickerPool) and map which deployments each EPP routes to:

kubectl get inferencepool -n $WVA_NS -o custom-columns=NAME:.metadata.name,SELECTOR:.spec.targetPortNumber 2>/dev/null || \
kubectl get endpointpickerpool -n $WVA_NS -o yaml 2>/dev/null

Determine which EPP routes to which decode deployments by inspecting the EPP's selector labels and matching them against the deployments' labels. Deployments served by the same EPP share a single WVA saturation-scaling ConfigMap — their thresholds (kv_cache_threshold, queue_length_threshold, etc.) must be identical. Deployments under different EPPs can have independent thresholds.