hqq-quantization

HQQ - Half-Quadratic Quantization

Fast, calibration-free weight quantization supporting 8/4/3/2/1-bit precision with multiple optimized backends.

When to use HQQ

Use HQQ when:

• Quantizing models without calibration data (no dataset needed)
• Need fast quantization (minutes vs hours for GPTQ/AWQ)
• Deploying with vLLM or HuggingFace Transformers
• Fine-tuning quantized models with LoRA/PEFT
• Experimenting with extreme quantization (2-bit, 1-bit)

Key advantages:

• No calibration: Quantize any model instantly without sample data
• Multiple backends: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference
• Flexible precision: 8/4/3/2/1-bit with configurable group sizes
• Framework integration: Native HuggingFace and vLLM support
• PEFT compatible: Fine-tune quantized models with LoRA

Use alternatives instead:

• AWQ: Need calibration-based accuracy, production serving
• GPTQ: Maximum accuracy with calibration data available
• bitsandbytes: Simple 8-bit/4-bit without custom backends
• llama.cpp/GGUF: CPU inference, Apple Silicon deployment

Quick start

Installation

pip install hqq

# With specific backend
pip install hqq[torch]      # PyTorch backend
pip install hqq[torchao]    # TorchAO int4 backend
pip install hqq[bitblas]    # BitBlas backend
pip install hqq[marlin]     # Marlin backend

Basic quantization

from hqq.core.quantize import BaseQuantizeConfig, HQQLinear
import torch.nn as nn

# Configure quantization
config = BaseQuantizeConfig(
    nbits=4,           # 4-bit quantization
    group_size=64,     # Group size for quantization
    axis=1             # Quantize along output dimension
)

# Quantize a linear layer
linear = nn.Linear(4096, 4096)
hqq_linear = HQQLinear(linear, config)

# Use normally
output = hqq_linear(input_tensor)

Quantize full model with HuggingFace

from transformers import AutoModelForCausalLM, HqqConfig

# Configure HQQ
quantization_config = HqqConfig(
    nbits=4,
    group_size=64,
    axis=1
)

# Load and quantize
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quantization_config,
    device_map="auto"
)

# Model is quantized and ready to use