write-tensorrt-plugin-integration-plan

Skill: Write a TensorRT Plugin Integration Plan

Purpose

Guide the agent through planning how a custom CUDA kernel will be wrapped as a TensorRT plugin so it can be invoked from inside a TensorRT engine — covering API choice (IPluginV3 vs IPluginV2DynamicExt), the plugin lifecycle, dynamic shape handling, serialization, mixed precision (FP16/INT8/FP8), workspace management, CUDA graph compatibility, and the C++/Python binding strategy. The output is an integration plan with explicit decisions, not the plugin source code itself.

Use this when

• A custom CUDA kernel must be deployed inside a TensorRT engine (production inference) and no built-in TRT layer covers it.
• An ONNX model contains an unsupported op that must be lowered to a custom plugin during ONNX → TRT conversion.
• A TRT-LLM-style codebase needs a new operator wrapped as a plugin to participate in engine builds.
• Migrating an existing IPluginV2DynamicExt plugin to IPluginV3 for a TensorRT 9.x or 10.x upgrade.

Do not use this when

• The operation is already covered by a built-in TRT layer (most GEMMs, conv shapes, softmax, layernorm fused into MHA, common activations) — the built-in path is almost always faster than a plugin because it participates in TRT's graph optimizer and tactic selection.
• The op can be expressed as a composition of existing TRT layers (e.g. IElementWiseLayer, IMatrixMultiplyLayer, IShuffleLayer) — TRT's optimizer fuses these well; an opaque plugin blocks fusion.
• The deployment target is not TensorRT. For vLLM use a vLLM custom op; for ONNX Runtime use an ORT custom op; for raw PyTorch use torch.library.
• The kernel relies on JIT autotuning at runtime (e.g. Triton with Python-side autotune). TRT engines are built ahead of time and cannot host Python or runtime JIT inside enqueue.
• The kernel allocates host memory or synchronizes inside the launch path — incompatible with CUDA graph capture, which is how production TRT inference servers reach low latency.