flashinfer

FlashInfer Skill

High-performance GPU kernel library for Large Language Model inference delivering state-of-the-art performance across diverse GPU architectures with optimized attention, GEMM, and MoE operations.

Official Sources:

• FlashInfer Documentation
• GitHub Repository
• Official Blog
• MLSys 2025 Paper (Best Paper Award)

What is FlashInfer?

Definition:

"A library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA."

Key Features:

• Unified APIs: Attention, GEMM, MoE with multiple backend implementations
• Paged & Ragged KV-Cache: Efficient memory management for dynamic batching
• Multi-Backend: FlashAttention-2/3, cuDNN, CUTLASS, TensorRT-LLM
• Quantization: FP8 and FP4 for attention, GEMM, MoE operations
• Production-Ready: CUDAGraph and torch.compile compatible
• Wide GPU Support: SM75 (Turing) through SM121 (Blackwell)

Quick Start

Installation

# Basic installation
pip install flashinfer-python

# With pre-compiled kernels (recommended)
pip install flashinfer-python flashinfer-cubin

# With JIT cache for specific CUDA version
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu129

System Requirements:

• Linux only
• Python 3.10-3.14
• CUDA 12.6, 12.8, 13.0, or 13.1
• GPU: Turing (T4) through Blackwell

Verify Installation

flashinfer show-config

Basic Usage

import torch
import flashinfer

# Single decode with paged KV-cache
output = flashinfer.single_decode_with_kv_cache(
    q=query,                    # (num_qo_heads, head_dim)
    kv_data=kv_cache,          # (num_pages, 2, num_kv_heads, page_size, head_dim)
    kv_indices=kv_page_indices,  # (num_pages,)
    kv_indptr=kv_page_indptr,    # (batch_size + 1,)
    kv_last_page_len=last_page_lengths,  # (batch_size,)
)