write-vllm-custom-op-integration-plan

Skill: Write a vLLM Custom Op Integration Plan

Purpose

Guide the agent through planning the integration of a custom CUDA or Triton kernel into the vLLM inference engine before any integration code is written — covering where the op plugs into the engine, paged KV cache and continuous batching compatibility, CUDA graph capture constraints, tensor parallelism implications, and the testing and benchmarking strategy. This skill produces an integration plan, not a kernel implementation.

Use this when

• A custom CUDA or Triton kernel exists (or is being designed) and the goal is to ship it inside vLLM rather than as a standalone library.
• Replacing an existing vLLM op (e.g., a faster RMSNorm, a fused activation, a new attention variant) and the model file has to be modified to call the new path.
• Adding a new quantization scheme that must be exposed through vLLM's LinearMethodBase and quantization config registry.
• The kernel needs to coexist with paged attention, continuous batching, CUDA graphs, and tensor parallelism without breaking any of them.

Do not use this when

• The kernel is being prototyped outside any serving engine; integration concerns are premature until correctness and standalone performance are established.
• vLLM already ships a competitive kernel for the same operation (FlashAttention-3, FlashDecoding, vllm-flash-attn, Marlin) and there is no novel reason to replace it. Integration cost is real and only justified by novel kernels, unsupported quantization formats, or custom serving features (constrained decoding, retrieval-augmented attention, RAG-fused ops).
• The kernel only helps in a regime that does not match vLLM's deployment profile (tiny batches, no KV cache, single-request offline). vLLM's throughput-oriented design will not surface the kernel's win.
• The user wants a kernel implementation. This skill is for the integration plan only.

Inputs the agent should gather first

• Op identity and intent: which op is being added or replaced? Is it a drop-in replacement (same numerics, faster), a new fusion (combining two existing ops), a new algorithm (e.g., a different attention variant), or a new quantization scheme?
• Target phase: prefill only, decode only, or both? Does the kernel handle chunked prefill (mixed prefill+decode in one batch)?
• Supported dtypes: fp16, bf16, fp8, int8, int4? Does it require a specific accumulation dtype?
• Quantization involvement: is the kernel itself quantized? Does it consume quantized weights, quantized activations, or both? What is the scale/zero-point layout?
• Tensor parallelism requirement: must the op work under TP=1, TP=2, TP=4, TP=8? Is the op TP-trivial (elementwise, normalization) or TP-sensitive (matmul, attention)?
• CUDA graph mode: must the kernel work under captured graphs (the default enforce_eager=False), or is eager-only acceptable? Eager-only severely limits production usefulness.
• vLLM version: v0 and v1 have very different code paths (v1 reworks the scheduler, model runner, and KV cache manager). Plans must target one explicitly.
• Hardware target: SM generations (sm_80, sm_89, sm_90, sm_100), AMD CDNA support requirement, and whether the kernel needs to ship on multiple architectures.
• Reference implementation: is there an existing PyTorch eager reference, a paper, or a public kernel to validate against?