llm-kv-cache-paged-attention

KV Cache & PagedAttention

Purpose

During decoding each new token attends to the keys and values of all previous tokens. Caching K and V avoids recomputing them, but the cache grows linearly with sequence length and batch size and usually becomes the binding constraint on concurrency. This skill gives the sizing math and the serving-engine levers (paging, prefix sharing, KV quantization, scheduling) to maximize throughput without OOM.

Agent Protocol

Trigger

"KV cache", "PagedAttention", "vLLM OOM", "gpu_memory_utilization", "max_num_seqs", "prefix caching", "long context memory", "KV cache quantization".

Input Context

• Model architecture: layers, KV heads, head dim (from config.json), dtype.
• GPU count/VRAM, target context length, expected concurrent requests.
• Prompt shape: shared system prompts or few-shot prefixes?

Output Artifact

KV budget calculation and serving configuration (vLLM flags or equivalent) with expected max concurrency.

Response Format

Model: {name} L={layers} kv_heads={h} head_dim={d} dtype={bytes}
KV per token: 2*L*h*d*bytes = {KiB}
KV budget: {GiB} -> tokens in cache ≈ {n}
Max concurrency at {ctx} tokens ≈ {n}
Config: --max-model-len {..} --gpu-memory-utilization {..} --kv-cache-dtype {..} --max-num-seqs {..}

Completion Criteria

• KV bytes/token computed from the real config (KV heads, not attention heads).
• Budget accounts for weights and activation/workspace memory.
• Concurrency estimate validated with a load test.
• Prefix caching considered for shared prompts.

Sizing Math

KV bytes per token = 2 (K and V) × num_layers × num_kv_heads × head_dim × bytes_per_element

Model (config)	Per-token KV (bf16)	8k-token sequence
Llama-3-8B: 32 layers, 8 KV heads, head_dim 128	2·32·8·128·2 = 128 KiB	1 GiB
Same model with fp8 KV cache	64 KiB	0.5 GiB
Hypothetical MHA (32 KV heads)	512 KiB	4 GiB

Grouped-query attention (GQA) shares each KV head across several query heads, cutting the cache by the group factor; multi-query attention (MQA) uses one KV head; multi-head latent attention (MLA, DeepSeek-V2/V3) stores a compressed latent instead of full K/V.