guidance

Guidance: Constrained LLM Generation

When to Use This Skill

Use Guidance when you need to:

• Control LLM output syntax with regex or grammars
• Guarantee valid JSON/XML/code generation
• Reduce latency vs traditional prompting approaches
• Enforce structured formats (dates, emails, IDs, etc.)
• Build multi-step workflows with Pythonic control flow
• Prevent invalid outputs through grammatical constraints

GitHub Stars: 18,000+ | From: Microsoft Research

Installation

# Base installation
pip install guidance

# With specific backends
pip install guidance[transformers]  # Hugging Face models
pip install guidance[llama_cpp]     # llama.cpp models

Quick Start

Basic Example: Structured Generation

from guidance import models, gen

# Load model (supports OpenAI, Transformers, llama.cpp)
lm = models.OpenAI("gpt-4")

# Generate with constraints
result = lm + "The capital of France is " + gen("capital", max_tokens=5)

print(result["capital"])  # "Paris"

Chat format with a local model

Constraint support requires local logit access. Regex, select(), and grammar-based constrained generation only work with local backends (Transformers, LlamaCpp). Remote API backends (OpenAI, and Azure variants) support unconstrained gen() / chat only — they cannot enforce token-level constraints. guidance 0.3.x has no models.Anthropic class.

from guidance import models, gen, system, user, assistant

# Local model (supports constrained generation)
lm = models.Transformers("microsoft/Phi-4-mini-instruct")

# Use context managers for chat format
with system():
    lm += "You are a helpful assistant."

with user():
    lm += "What is the capital of France?"

with assistant():
    lm += gen(max_tokens=20)

Core Concepts

1. Context Managers

Guidance uses Pythonic context managers for chat-style interactions.

from guidance import system, user, assistant, gen

lm = models.Transformers("microsoft/Phi-4-mini-instruct")

# System message
with system():
    lm += "You are a JSON generation expert."

# User message
with user():
    lm += "Generate a person object with name and age."

# Assistant response
with assistant():
    lm += gen("response", max_tokens=100)

print(lm["response"])