vllm

vLLM v0.19.0 Python API Guide

Comprehensive reference for vLLM -- a high-throughput, memory-efficient inference engine for large language models.

Overview

vLLM provides two main interfaces:

1. Offline inference via the LLM class -- batch processing with automatic memory management
2. Online serving via vllm serve -- OpenAI-compatible REST API with streaming

Key capabilities:

• High-throughput batched generation with PagedAttention
• OpenAI-compatible chat completions, completions, and embeddings API
• Structured outputs (JSON schema, regex, grammar, choice)
• LoRA adapter hot-swapping
• Multimodal inputs (images, audio, video)
• Tensor, pipeline, and data parallelism
• Quantization (AWQ, GPTQ, FP8, and more)
• Prefix caching for shared prompt prefixes
• Tool calling / function calling

Quick Start

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
params = SamplingParams(temperature=0.8, max_tokens=256)

# Text completion (no chat template)
outputs = llm.generate(["The future of AI is"], params)
print(outputs[0].outputs[0].text)

# Chat completion (applies chat template)
outputs = llm.chat(
    [{"role": "user", "content": "What is vLLM?"}],
    sampling_params=params,
)
print(outputs[0].outputs[0].text)

When to Use This Skill

• Writing Python code that uses vLLM for inference
• Configuring vllm serve for production deployment
• Using structured outputs (JSON, regex, grammar)
• Setting up LoRA adapter serving
• Passing multimodal inputs (images, audio, video)
• Tuning sampling parameters
• Understanding output types and result handling
• Migrating from vLLM v0.16.x to v0.19.0

Core Concepts

LLM Class

The main offline inference API. Created with a model name/path, automatically manages GPU memory and batching.

from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

SamplingParams

Controls generation behavior: temperature, top-p, max tokens, stop sequences, structured outputs, etc.