LM Format Enforcer

Common errors

• ImportError: cannot import name 'LogitsWarper' from 'transformers.generation.logits_process'

  cause This error typically occurs due to version incompatibility between `lm-format-enforcer` and the `transformers` library, where `LogitsWarper` or related classes have been moved or renamed in newer `transformers` versions.
  fix Upgrade `lm-format-enforcer` to its latest version (e.g., `pip install --upgrade lm-format-enforcer`) or downgrade `transformers` to a compatible version, as specified in `lm-format-enforcer`'s documentation or release notes.

• new_logits_warper() got an unexpected keyword argument 'device'

  cause This error indicates that the `lm-format-enforcer`'s integration with `transformers` is passing an unsupported argument ('device') to a `transformers` function, often due to an API change in `transformers` or an outdated `lm-format-enforcer` version.
  fix Update `lm-format-enforcer` to the latest version to ensure compatibility with recent `transformers` releases (e.g., `pip install --upgrade lm-format-enforcer`). If the issue persists with the latest versions, consult the library's GitHub issues for specific compatibility workarounds.

• lm-format-enforcer package not found. please run `pip install lm-format-enforcer`

  cause The `lm-format-enforcer` library has not been installed in the current Python environment, or the environment where the code is being run does not have access to the installed package.
  fix Install the library using pip: `pip install lm-format-enforcer`. If running in a specific environment (e.g., virtual environment, Docker), ensure the installation is performed within that environment.

• LM Format Enforcer producing incomplete JSON

  cause Despite using `lm-format-enforcer` with a JSON schema, the generated output from the LLM is often truncated or malformed, failing to adhere to the complete JSON structure. This can happen if the model is struggling to follow the prompt and the constraints, or if the schema itself has complexities like array constraints that the enforcer might struggle with in older versions.
  fix Improve the prompt to explicitly guide the LLM towards generating valid JSON, use a simpler JSON schema if possible, and ensure `lm-format-enforcer` is updated to the latest version to benefit from bug fixes and improved schema support. For complex schemas, consider simplifying them or explicitly mentioning challenging aspects in the prompt.

• Unknown LMFormatEnforcer Problem: File "/opt/homebrew/lib/python3.10/site-packages/lmformatenforcer/tokenen..." (when using Union types with Pydantic)

  cause This specific traceback points to an issue within `lm-format-enforcer`'s internal token enforcer when processing complex Pydantic types, particularly `Union` types, indicating that the library might not fully support all Pydantic features in certain versions.
  fix Upgrade `lm-format-enforcer` to the latest version, as support for more Pydantic features is continuously being added. If the issue persists, try simplifying the Pydantic model by avoiding complex `Union` types or restructuring the schema to use more basic types if possible.

Warnings

• breaking The return type of `TokenEnforcer.get_allowed_tokens()` changed in v0.11.1 to be torch tensor bitmask based for vLLM V1 integration. This is a breaking change if you directly call or rely on the return type of this internal function.
  Fix: Avoid direct calls to `TokenEnforcer.get_allowed_tokens()`. If you need similar functionality, check the latest integration patterns or raise a GitHub issue. For vLLM integration, ensure `use_bitmask=True` is handled if customizing.
• gotcha The primary user-facing API for integrating `lm-format-enforcer` with LLMs (e.g., Transformers, vLLM) involves specific `build_..._prefix_allowed_tokens_fn` functions, rather than directly instantiating a generic `LMFormatEnforcer` class. This is a common point of confusion for new users.
  Fix: Refer to the library's examples and documentation for the correct integration pattern, typically importing `build_transformers_prefix_allowed_tokens_fn` or `build_vllm_prefix_allowed_tokens_fn` from their respective `integrations` submodules.
• gotcha The library has specific Python version requirements (`>=3.8, <4.0`). Using unsupported Python versions may lead to unexpected errors or installation issues.
  Fix: Ensure your environment uses a compatible Python version (e.g., Python 3.8, 3.9, 3.10, 3.11).
• gotcha Compatibility with `transformers` and `pydantic` libraries can be strict. Older versions of `transformers` (pre-4.38.0) and `pydantic` (pre-2.0.0) might cause issues or not be fully supported.
  Fix: Always use the officially recommended or latest compatible versions of `transformers` (>=4.38.0) and `pydantic` (>=2.0.0,<3.0.0) as specified in the `pyproject.toml` or `setup.py`.
• gotcha When integrating with vLLM, ensure your vLLM version is compatible with the `lm-format-enforcer` version. Specific vLLM versions (e.g., vLLM V1) may require particular `lm-format-enforcer` versions and features like the `use_bitmask` flag.
  Fix: Consult the `lm-format-enforcer` GitHub README and release notes for the recommended vLLM version and any specific flags or configuration needed for stable integration.

Install

• pip install lm-format-enforcer Standard installation
• pip install lm-format-enforcer[vllm] Installation with vLLM support

Imports

• JsonSchemaParser

  from lm_format_enforcer.json_schema_parser import JsonSchemaParser

• RegexParser

  from lm_format_enforcer.regex_parser import RegexParser

• build_transformers_prefix_allowed_tokens_fn
  wrong:

  from lm_format_enforcer import LMFormatEnforcer

  correct:

  from lm_format_enforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn

  The core 'LMFormatEnforcer' class is usually an internal component; users interact with integration-specific builder functions like this one, combined with a parser.
• build_vllm_prefix_allowed_tokens_fn
  wrong:

  from lm_format_enforcer import VLLMFormatEnforcer

  correct:

  from lm_format_enforcer.integrations.vllm import build_vllm_prefix_allowed_tokens_fn

  Similar to transformers integration, vLLM users should import the builder function rather than a direct enforcer class.

Quickstart

This quickstart demonstrates how to enforce a JSON Schema output using a Hugging Face Transformers model. It initializes a tokenizer and model, defines a JSON schema, creates a `JsonSchemaParser`, and then uses `build_transformers_prefix_allowed_tokens_fn` to generate text that strictly adheres to the defined format.

from transformers import AutoTokenizer, AutoModelForCausalLM
from lm_format_enforcer.json_schema_parser import JsonSchemaParser
from lm_format_enforcer.integrations.transformers import build_transformers_prefix_allowed_tokens_fn
import torch

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

# Define the JSON schema
json_schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 0},
        "isStudent": {"type": "boolean"}
    },
    "required": ["name", "age", "isStudent"]
}

# Create the parser
json_parser = JsonSchemaParser(json_schema)

# Build the prefix_allowed_tokens_fn for transformers integration
prefix_allowed_tokens_fn = build_transformers_prefix_allowed_tokens_fn(tokenizer, json_parser)

prompt = "Please generate a JSON object describing a person with name, age, and student status:\n"

# Encode the prompt
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Generate text with format enforcement
# GPT2 might not perfectly follow instructions but the *format* will be enforced.
output = model.generate(
    input_ids,
    max_new_tokens=100,
    prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
    pad_token_id=tokenizer.eos_token_id,
    do_sample=False, # For deterministic generation where possible
    num_beams=1
)

# Decode and print the result
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
# Example output (format enforced):
# {"name": "Alice", "age": 25, "isStudent": true}