mdc-vllm

vLLM Best Practices

vLLM is the gold standard for high-throughput LLM inference. Adhere to these guidelines to maximize performance, ensure reproducibility, and maintain robust LLM services.

1. Environment & Installation

Always use isolated conda environments and pin vLLM to your CUDA version. This prevents binary incompatibilities and ensures reproducible deployments.

❌ BAD:

# Unreliable global install, prone to CUDA/PyTorch mismatches
pip install vllm

✅ GOOD:

# For NVIDIA GPUs (CUDA 12.1 is default, check vLLM docs for current default)
conda create -n vllm_env python=3.10 -y
conda activate vllm_env
pip install vllm==0.5.2 # Pin the exact version in requirements.txt

# For NVIDIA GPUs (CUDA 11.8 specific version, e.g., v0.4.0)
conda create -n vllm_cu118 python=3.10 -y
conda activate vllm_cu118
export VLLM_VERSION=0.4.0
export PYTHON_VERSION=310
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118

• Action: Ensure requirements.txt explicitly lists vllm==X.Y.Z.
• Hardware: Target GPUs with compute capability ≥ 7.0 (V100, A100, H100, etc.).

2. Code Organization & Structure

Separate inference logic from data preprocessing and business logic. Use a modular approach for clarity and testability.

❌ BAD:

# main.py - monolithic script, hard to test or scale
from vllm import LLM, SamplingParams
# ... data loading, preprocessing, model init, inference, postprocessing ...

✅ GOOD:

# llm_service/inference_engine.py
from vllm import LLM, SamplingParams
from typing import List

class InferenceEngine:
    def __init__(self, model_path: str, **kwargs):
        """Initializes the vLLM engine with specified model and configurations."""
        self.llm = LLM(model=model_path, **kwargs)
        self.sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)

    def generate(self, prompts: List[str]) -> List[str]:
        """Generates responses for a list of prompts using configured sampling parameters."""
        outputs = self.llm.generate(prompts, self.sampling_params)
        return [output.outputs[0].text for output in outputs]

# llm_service/main.py (or API endpoint, e.g., with FastAPI)
from .inference_engine import InferenceEngine
from typing import Dict

def serve_llm_request(request_data: Dict) -> Dict:
    """Handles an incoming LLM request, orchestrating preprocessing, inference, and postprocessing."""
    # 1. Preprocessing (e.g., validate input, format prompt from request_data)
    prompts = [request_data["text"]] # Simplified example
    
    # 2. Inference
    # Model path and parallelism should be loaded from config, not hardcoded here
    engine = InferenceEngine(model_path="mistralai/Mistral-7B-Instruct-v0.2",
                             tensor_parallel_size=2) # Explicitly configure parallelism
    results = engine.generate(prompts)
    
    # 3. Postprocessing (e.g., format output for API response, add metadata)
    return {"generated_text": results[0]}

• Action: Define LLM engine parameters explicitly (e.g., tensor_parallel_size for distributed inference).