vllm-gemma-4-31b

Gemma 4 31B on vLLM — operating-point reference

One model, measured. This is a worked operating point, not the general method. The reusable machinery lives in the vllm plugin: flag and env semantics in vllm-configuration, the K8s/container manifest in vllm-deployment, reproducing or re-measuring these numbers in vllm-benchmarking, and the prompt-side template in vllm-chat-templates. Deploying a different model? Use those, not this.

For platform engineers deploying google/gemma-4-31B-it (BF16, FP8) or its community quants (e.g. cyankiwi/gemma-4-31B-it-AWQ-4bit, RedHatAI/*-Gemma-4-31B-*) on vLLM 0.20–0.25.1. Pulls together measurements from a Verda 2× H100 SXM5 80GB audit on 2026-04-30 — taken on vLLM 0.20.0 and not re-run since (see references/bench-numbers.md; the shape of the curves has held, but treat the absolute figures as 0.20.0 observations) — and the upstream constraints that shape the answer.

Version ceiling (re-verified 2026-08-11): hold at vLLM 0.25.1. Do not take 0.26.0, 0.27.0 or 0.27.1. All three gemma-4-relevant regressions are still OPEN, and every fix PR for them is still unmerged, so neither 0.27.0 (2026-08-10) nor the 0.27.1 patch (2026-08-11 — a single change, #50424, scoped to Qwen3DSparkModel, nothing Gemma) clears any of them: #49955 (trailing <turn|> leaked into output — not spec-decode-specific after all, see the pitfall below; fix PR #50964 open, and #50263 was tested by the reporter and did not fix it), #50477 (gemma4 parser silently ignores named forced tool_choice; a second reporter extends it to tool_choice: "required", which returns finish_reason: "tool_calls" with prose in content and tool_calls: null — fix PR #51524 open), #50159 (Model Runner V2 over-reports available KV → CUDA OOM under saturating load; crashes earlier with EAGLE — no fix PR at all). Two independent A/B runs on #50159 now localise it: the gap is CUDA-graph capture headroom, not KV capacity. MRv1 reserves ~0.65 GiB for capture where MRv2 reserves ~0.12 GiB, and the whole discrepancy disappears under --enforce-eager. Severity scales with model size and inversely with card size.

The one thing 0.27.0 does add for this model is the ViT CUDA graph (#46837, merged 2026-07-25, listed in the v0.27.0 release notes): full SupportsEncoderCudaGraph for Gemma4ForConditionalGeneration, making the vision encoder 100% statically compiled by replacing the pooler's data-dependent slicing with a fixed-shape gather. Relevant only to multimodal traffic, and not worth taking the three regressions for — re-evaluate when #49955 and #50159 close.