bitsandbytes

BitsAndBytes: Quantization for LLMs

Overview

BitsAndBytes provides quantization methods for efficient LLM loading and inference, including 8-bit (LLM.int8()) and 4-bit (NF4/FP4) quantization. Apply this skill for memory-efficient model loading, quantization-aware training, and running large models on limited GPU resources.

When to Use This Skill

This skill should be used when:

• Loading large language models in 8-bit or 4-bit
• Reducing GPU memory usage for inference
• Running 7B+ models on consumer GPUs
• Fine-tuning quantized models with PEFT
• Implementing mixed-precision quantization
• Optimizing inference latency
• Quantization-aware training
• Using NF4 format for better quality

Quick Start

Basic Import and Setup

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from bitsandbytes import BitsAndBytesConfig

8-bit Quantization

# Configure 8-bit quantization
quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,  # Threshold for outlier detection
    llm_int8_has_fp16_weight=False,  # Keep fp16 for some weights
)

# Load model with 8-bit quantization
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=quantization_config,
    device_map="auto",
)

# Use model normally
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
inputs = tokenizer("Hello, my name is", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))

4-bit Quantization (NF4)

# Configure 4-bit NF4 quantization
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # NF4 format for better quality
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,  # Double quantization for more compression
)

# Load model with 4-bit quantization
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=quantization_config,
    device_map="auto",
)

# Generate
outputs = model.generate(**inputs)