agentsop-vllm

vLLM Serving SOP

1. 何时激活 (When to activate)

Activate this skill when any of the following hold:

• The user wants to serve an LLM in production (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure.
• The user is comparing inference engines (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama).
• The user reports a vLLM operational issue: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice.
• The user is sizing hardware for an open-weights model (Llama / Qwen / Mixtral / DeepSeek-V3) and asking about tensor/pipeline parallelism.
• The user mentions PagedAttention, prefix caching, continuous batching, chunked prefill, or speculative decoding.

Do NOT activate for: training/fine-tuning (use accelerate/deepspeed/trl), CPU-only edge inference (use llama.cpp/Ollama), Apple Silicon production (vLLM Metal/MPS is experimental, not production-ready as of 2026) [aimadetools.com 2026], API-only consumption of hosted models (just call the OpenAI/Anthropic SDK).

---

2. 核心心智模型 (Core Mental Model)

2.1 The OS abstraction: KV cache as virtual memory

vLLM's defining insight (Kwon et al., SOSP 2023) is that LLM serving's bottleneck was not compute — it was KV-cache memory fragmentation. Pre-vLLM systems pre-allocated a contiguous KV-cache slot per request, sized for the maximum possible output length; in early 2023, inference engines used only 20–40% of available GPU memory because of internal+external fragmentation [arxiv.org/abs/2309.06180; zilliz.com/learn].

PagedAttention applies classic OS paging to KV cache:

• Block = fixed-size chunk of KV cache (default 16 tokens; ~12.8 KB for a 13B model) [medium.com/@mandeep0405].
• Logical blocks per request → block table → physical blocks in GPU memory (analogous to virtual→physical page table).
• Blocks need not be contiguous. The attention kernel reads scattered physical blocks via the block table and presents them as a logical contiguous sequence.
• Copy-on-write + prefix sharing: multiple requests that share a prefix (e.g. a system prompt, few-shot examples) share KV blocks; a write triggers a per-request copy [arxiv.org/abs/2309.06180].