write-int8-quantized-kernel

Skill: Write an INT8 Quantized Kernel

Purpose

Guide the agent through designing and implementing an INT8 quantized matrix multiplication or linear layer kernel for inference, covering quantization scheme selection, scale computation, int32 accumulation, dequantization epilogue, and the decision between custom code and library solutions.

Use this when

• Writing an inference kernel that needs to reduce memory bandwidth and/or increase throughput by operating on INT8 weights and activations.
• Implementing a quantized linear layer where scales and zero-points are known at kernel call time (static or dynamic).
• The hardware is Turing (sm_75) or later, where the dp4a INT8 dot product instruction is available.
• Evaluating whether a custom INT8 GEMM is justified versus using cuBLAS INT8 (cublasGemmEx with CUDA_R_8I) or CUTLASS INT8 GEMM templates.

Do not use this when

• The target hardware is pre-Turing (sm_70 or earlier) — there is no hardware INT8 dot product instruction and the implementation falls back to emulation, which is unlikely to outperform fp16.
• The workload is training (not inference) — INT8 gradients require careful handling that is a separate design problem.
• The accuracy budget cannot tolerate INT8 quantization error. Evaluate accuracy first before committing to INT8.
• cuBLAS cublasGemmEx with CUDA_R_8I inputs and CUDA_R_32I compute already satisfies the performance requirement. Use the library unless there is a specific reason (custom epilogue, non-standard quantization scheme, latency constraints) that the library does not meet.

Inputs the agent should gather first

• Quantization granularity: per-tensor (one scale for the entire tensor), per-channel (one scale per output channel of the weight), or per-token (one scale per token/row of the activation). This determines the dequantization operation.
• Symmetric vs asymmetric quantization: symmetric maps the range [-127, 127] to [-max_abs, max_abs] with zero-point = 0. Asymmetric maps [0, 255] (or [-128, 127]) to an arbitrary range with a non-zero zero-point. Symmetric is strongly preferred for inference because zero-point handling adds extra arithmetic to the hot loop.
• Accumulation dtype: INT32 is mandatory. INT8 accumulation overflows for any practical dot product length.
• Output dtype: FP32 or FP16 after dequantization. This determines the epilogue.
• Scale dtype and location: are scales stored as fp32 scalars, fp32 tensors, or fp16? Are they computed offline (static quantization) or at kernel launch time (dynamic quantization)?
• Hardware target: compute capability (sm_75 for Turing dp4a, sm_80 for Ampere, sm_89 for Ada).
• Weight layout: are weights already quantized and stored as INT8? Are they in row-major or column-major order?
• Activation layout: contiguous rows? Are activations quantized offline or online (at each forward pass)?