llama-cpp-python: Python Bindings for llama.cpp

0.3.20 · active · verified Tue Apr 14

Python bindings for the `llama.cpp` library, enabling efficient local inference of large language models (LLMs) on various hardware, including CPUs and GPUs (NVIDIA, Apple Metal, AMD ROCm). It provides both a high-level API for easy model interaction and a low-level API for direct C API access. The library is actively maintained with frequent updates, often mirroring upstream `llama.cpp` changes, and currently stands at version 0.3.20.

Warnings

breaking Installation with GPU acceleration (CUDA, Metal, ROCm) often requires setting specific `CMAKE_ARGS` environment variables or using pre-built wheels from custom index URLs, along with correctly configured C++ compilers and GPU toolkits. Default `pip install` typically provides CPU-only support.
Fix: Refer to the official documentation or GitHub README for specific `CMAKE_ARGS` and installation instructions for your desired backend (e.g., `CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python`). Ensure all necessary C++ compilers and GPU toolkits are correctly installed and configured in your system's PATH.
gotcha On Apple Silicon (M1/M2) Macs, `llama-cpp-python` can default to building an x86 version if an ARM64 Python interpreter is not used. This results in significantly slower performance.
Fix: Ensure you are using an ARM64 version of Python (e.g., from Miniforge). You might need to explicitly specify `CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DGGML_METAL=on"` during installation.
breaking The library closely tracks upstream `llama.cpp` development, which can introduce breaking changes to the underlying C API that may propagate to the Python bindings. Notable past changes include the transition to GGUF model format and changes in KV cache management functions.
Fix: Always check the changelog when upgrading `llama-cpp-python`. If upgrading, use `--upgrade --force-reinstall --no-cache-dir` to ensure a clean rebuild from source. Be prepared for potential API adjustments, particularly in low-level usage or integrations with other libraries.
gotcha LLM models require specific chat formats (e.g., 'llama-2', 'chatml') for proper conversational interaction. Using the wrong format can lead to 'weird' or unparsable responses, especially in chat completion APIs.
Fix: When initializing `Llama` for chat, explicitly set the `chat_format` parameter (e.g., `llm = Llama(..., chat_format="llama-2")`). Consult the model card or `llama-cpp-python` documentation for the correct format for your chosen model.
gotcha To generate embeddings, you must explicitly enable them by passing `embedding=True` to the `Llama` constructor when initializing the model.
Fix: Initialize your model with `llm = Llama(model_path="...", embedding=True)`. Then use `llm.create_embedding(...)`.

Install

pip install llama-cpp-python Basic CPU support (builds from source)
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu Basic CPU support (pre-built wheel)
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python With NVIDIA CUDA (requires CUDA Toolkit & C++ compiler)
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python With Apple Metal GPU (macOS, requires arm64 Python)
CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python With AMD ROCm/hipBLAS (requires ROCm toolkit)
CMAKE_ARGS="-DLLAMA_OPENBLAS=on" pip install llama-cpp-python With OpenBLAS (optimized CPU)

Imports

Llama
```
from llama_cpp import Llama
```
This is the primary high-level class for interacting with loaded models.
LlamaGrammar
```
from llama_cpp import LlamaGrammar
```
Used for constrained text generation with GBNF grammars (e.g., JSON output).
LlamaHFTokenizer
```
from llama_cpp import LlamaHFTokenizer
```
Required for certain models (like Functionary v2) where HuggingFace tokenizers are needed due to discrepancies with `llama.cpp`'s default tokenizer.

Quickstart

This quickstart demonstrates how to load a GGUF model and generate text using the high-level `Llama` class. It highlights important parameters like `model_path`, `n_ctx` for context size, and `n_gpu_layers` for GPU offloading. Ensure your model is in the GGUF format.

import os
from llama_cpp import Llama

# Ensure you have a GGUF model downloaded, e.g., to a 'models' directory.
# Example: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf
model_path = os.environ.get('LLAMA_MODEL_PATH', './models/llama-2-7b-chat.Q4_K_M.gguf')

# Initialize the Llama model
# Set n_gpu_layers to a value > 0 for GPU acceleration (requires GPU install config)
llm = Llama(
    model_path=model_path,
    n_ctx=2048,  # Context window size
    n_gpu_layers=0, # Set to > 0 for GPU, -1 to offload all layers if GPU is available
    verbose=False  # Suppress llama.cpp verbose output
)

# Generate a completion
prompt = "Q: Name the planets in the solar system? A: "
output = llm(prompt, max_tokens=128, stop=["Q:", "\n"], echo=True)

print(output["choices"][0]["text"])

view raw JSON →