TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Skill Claude CodeCodex
Debug AutoDeploy accuracy regressions vs a reference score (PyTorch backend or published baseline). Use when an AutoDeploy model's eval score is significantly below the reference and the root cause is unknown.
Skill Claude CodeCodex
Claude Code skill (trtllm-agent-toolkit): implement or extend TensorRT-LLM AutoDeploy fusion transforms under transform/library/ in a TensorRT-LLM checkout. Prefer existing kernels and custom ops; use Triton only when no viable existing-kernel path exists. Use ad-graph-dump for ADDUMPGRAPHSDIR workflows. Covers…
Skill Claude CodeCodex
Check whether AutoDeploy YAML configs were actually applied by analyzing server logs and optionally graph dumps (ADDUMPGRAPHSDIR). Use when the user wants to verify config application, debug config issues, or check if AutoDeploy transforms (piecewise CUDA graph, multi-stream, sharding, fusion, etc.) were applied or…
Skill Claude CodeCodex
Enable and interpret TensorRT-LLM AutoDeploy FX graph text dumps via ADDUMPGRAPHSDIR. Use when you need before/after graphs per transform, to locate subgraphs, or to confirm a rewrite ran. Paths and behavior are grounded in tensorrtllm/torch/autodeploy (GraphWriter, BaseTransform). Complements…
Skill Claude CodeCodex
Visualize a specific transformer decoder layer from an AutoDeploy FX graph text dump as a hierarchical DOT/PNG diagram. Optionally annotate nodes with actual GPU kernel names and durations from an nsys trace. Use when the user wants to visualize, inspect, or debug a layer in an AutoDeploy model graph dump. Triggers…
Skill Claude CodeCodex
Translates a HuggingFace model into a prefill-only AutoDeploy custom model using reference custom ops, validates with hierarchical equivalence tests.
Skill Claude CodeCodex
Adds sharding-aware IR hints (op substitutions, sharding kwargs, allreduce insertions) directly into an existing AutoDeploy custom model (modeling.py). Edits the file in place — no separate ir.py copy. Validates with applyshardinghints and end-to-end multi-GPU runs.
Skill Claude CodeCodex
Check the local execution environment for GPU availability, Docker support, and Slurm access. Returns the execution scenario (satisfied, local, docker, satisfied, local, direct, satisfied, slurm, local, or notsatisfied), the number of available GPUs, and the GPU type. On Slurm login nodes without local GPUs, the…
Skill Claude CodeCodex
Compile TensorRT-LLM on a compute node inside a Docker container. Use this when already on a compute node with GPUs visible.
Skill Claude CodeCodex
Execute a TensorRT-LLM workload locally in Docker. Runs a fully-resolved Docker command in background, monitors completion, reads logs, and reports results. Workflow-agnostic — does not need to know if the workload is pytest, eval, benchmark, or a custom script.
Skill Claude CodeCodex
Submit and monitor a Slurm job on a local cluster. Supports two modes: (1) Persistent allocation (default) — allocates nodes once via nohup salloc, imports the container once, installs once, and reuses across runs by setting SLURM env vars and running the sbatch script via bash. (2) One-shot sbatch — submits a…
Skill Claude CodeCodex
Remote SLURM cluster development via SSH. Use when running jobs, profiling, or developing on a remote SLURM cluster with pyxis/enroot containers. Covers SSH connection management, srun/sbatch/salloc job patterns, tmux-based allocation persistence, file transfer, and safe remote file access. Works with any SLURM…
Skill Claude CodeCodex
Compile TensorRT-LLM on a SLURM cluster. Covers submitting a batch job with a container image, monitoring the job, and verifying the build. Use when the user wants to compile TRT-LLM remotely via SLURM rather than on a local compute node.
Skill Claude CodeCodex
Write and implement GPU kernels using NVIDIA CuTe DSL (CUTLASS 4.x Python API) — NOT for Triton, CUDA C++, or conceptual explanations. Trigger only when the user wants to write or implement a kernel, not when asking questions about CuTe DSL concepts or layouts. CuTe DSL uses cute.jit/cute.kernel decorators and…
Skill Claude CodeCodex
Optimize existing Triton kernels for NVIDIA TileIR backend on Blackwell GPUs (sm100+). Adds TileIR-specific autotune configs: occupancy, numctas, TMA descriptors. Covers kernel classification (dot-related, norm-like, elementwise, reduction), type-specific transformations, and PTX-vs-TileIR benchmarking. Triggered by…
Skill Claude CodeCodex
ONLY for OpenAI Triton (@triton.jit) kernel development. NEVER use for CUDA C++ kernels, TileIR, or profiling tools (ncu, nsys). The user's request must involve Triton explicitly. Covers Triton-specific patterns: fused elementwise, reductions (softmax, LayerNorm, RMSNorm), tiled GEMM with triton.autotune, and flash…
Skill Claude CodeCodex
Performance analysis coordination workflow. Guides profiling delegation, bottleneck classification (compute/memory/launch/communication/sync), and structured report generation. Use when the user asks to analyze performance, profile a workload, check MFU/SOL, or diagnose bottlenecks.
Skill Claude CodeCodex
Analyze host/CPU overhead in TensorRT-LLM inference from nsys traces. Detect whether host overhead is the bottleneck using GPU idle ratio, host prep exposed ratio, and per-phase evidence. For regressions, isolate forward steps via allreduce/NVTX patterns, compare host operation breakdowns across versions, and identify…
Skill Claude CodeCodex
Profiles and optimizes TensorRT-LLM host/CPU overhead using lineprofiler (with nsys support planned). Runs iterative profile-analyze-optimize-validate rounds. Use when GPU utilization is low or optimizing PyExecutor throughput.
Skill Claude CodeCodex
Analyze ncu (NVIDIA Nsight Compute) profiling output: SOL% bottleneck classification, roofline analysis, occupancy diagnosis, memory hierarchy analysis, warp stall analysis, metric interpretation, and programmatic .ncu-rep report analysis. NOT for kernel writing or code generation, Nsight Systems (nsys), host-side…
Skill Claude CodeCodex
Nsight Systems (nsys) CLI for system-level timeline profiling. Use when the user wants to run nsys profile, analyze .nsys-rep reports, use nsys stats/analyze/recipe commands, diagnose GPU idle time from timeline traces, or profile distributed training with NCCL overlap analysis. NOT for kernel-level metrics like SOL%…
Skill Claude CodeCodex
Casebook of past successful and classic TensorRT-LLM optimizations (runtime/execution and kernel level) recorded as reusable decision precedents. Consult when deciding which optimization to apply for a classified bottleneck or a given config/model/hardware, to find prior art and adapt a proven approach instead of…
Skill Claude CodeCodex
Performance optimization coordination playbook. Contains specialist routing table, TileIR two-step pipeline, kernel generation specialist selection, prioritization criteria, and safe modification workflow. Use when the user asks to apply optimizations, write kernels, or improve performance. Covers both user-specified…
Skill Claude CodeCodex
Apply CUDA Graphs to PyTorch workloads — API selection (torch.compile, PyTorch makegraphedcallables, TE makegraphedcallables, MCore CudaGraphManager, FullCudaGraphWrapper, manual torch.cuda.graph), code compatibility, capture workflows, dynamic pattern handling, and troubleshooting. Triggers: CUDA graph…