Guidance

0.3.1 · active · verified Sat Feb 28

Microsoft-backed constrained generation framework for LLMs. Programs interleave control flow with generation via lm += gen(...) syntax. Supports regex, CFGs, JSON schema, and select() constraints using a Rust-based llguidance engine. Backends: Transformers, llama.cpp, OpenAI, Azure AI. Model objects are immutable — each += produces a copy.

Warnings

breaking guidance.llms namespace removed. All pre-0.1.x code using guidance.llms.OpenAI() or guidance.llm.OpenAI() raises AttributeError.
Fix: Use from guidance.models import OpenAI (or Transformers, LlamaCpp, AzureOpenAI).
breaking bench module removed in 0.2.x. from guidance import bench raises ImportError.
Fix: JSON schema benchmarking moved to external repo guidance-ai/guidance-bench.
breaking @guidance decorator requires explicit stateless=True for pure grammar composition functions that do not call gen() internally.
Fix: Annotate stateless functions as @guidance(stateless=True). Pre-0.2 cookbook examples omit this and produce incorrect behavior.
breaking llama-cpp-python version must match guidance pin (currently >=0.3.12). Mismatched versions cause AttributeError or silent failures on LlamaCpp model init.
Fix: pip install 'llama-cpp-python>=0.3.12' explicitly. Check guidance release notes for updated pin on each upgrade.
gotcha Model objects are immutable. lm += ... does not mutate in place — it returns a new copy. Not storing the result silently discards generated output.
Fix: Always assign: lm = lm + ... or use lm += ... inside a with block where lm is re-bound.
gotcha pip install guidance does not install any inference backend. Importing Transformers or LlamaCpp without the backend raises ImportError at runtime.
Fix: Install the required backend separately: pip install transformers or pip install llama-cpp-python.
gotcha OpenAI and Azure backends do not support token-level constrained decoding (regex/CFG). Constraints fall back to prompt-based steering only. Full constrained generation requires a local backend.
Fix: Use Transformers or LlamaCpp backend for hard constraints. OpenAI backend supports JSON schema via OpenAI structured outputs API only.

Install

pip install guidance Python (base)
pip install guidance[transformers] Python (Transformers backend)
pip install guidance[llamacpp] Python (llama.cpp backend)

Imports

models.OpenAI
```
from guidance.models import OpenAI
```
guidance.llms namespace removed entirely. guidance.llm (singular) also removed. Only guidance.models is valid.
Transformers
```
from guidance.models import Transformers
```
Import directly from guidance.models. Requires transformers package installed separately.

Quickstart

Minimal constrained generation with Transformers backend. Model object is immutable — lm is a copy at each step.

from guidance import system, user, assistant, gen
from guidance.models import Transformers

lm = Transformers('microsoft/Phi-4-mini-instruct')

with system():
    lm += 'You are a helpful assistant'
with user():
    lm += 'What is the capital of France?'
with assistant():
    lm += gen(name='answer', max_tokens=20)

print(lm['answer'])

view raw JSON →