LlamaIndex Ollama LLM Integration

0.10.1 · active · verified Wed Apr 15

The `llama-index-llms-ollama` library provides an integration for LlamaIndex to utilize Large Language Models (LLMs) hosted locally via Ollama. It enables users to leverage various open-source models (like Llama, Mistral, Gemma, Phi-3, etc.) for tasks such as completions and chat within a LlamaIndex application, without relying on cloud-based LLM services. The current version is 0.10.1, released on March 20, 2026, and follows LlamaIndex's active and rapid release cadence for its integration packages.

Warnings

gotcha Ollama Server Prerequisite: The Ollama application must be installed and actively running on your local machine, and the desired LLM model (e.g., `llama3.1`) must be pulled using `ollama pull <model_name>` before this integration can connect to it.
Fix: Install Ollama from https://ollama.ai/, run `ollama serve` in your terminal, and pull your chosen model (e.g., `ollama pull llama3.1`).
gotcha Default Timeout: The default request timeout (often 30 seconds) may be too short for larger local LLMs or slower machines, leading to `Timeout` errors.
Fix: Increase the `request_timeout` parameter when initializing `Ollama`: `llm = Ollama(..., request_timeout=120.0)`.
gotcha High Memory Usage: Running large local LLMs (e.g., Llama 3.1 8B) through Ollama can be memory-intensive, often requiring 32GB of RAM or more, especially when combined with embedding models.
Fix: Monitor system RAM usage. Consider using smaller, quantized models (e.g., `llama3.1:7b-q4_0`), or adjust the `context_window` parameter in the `Ollama` constructor to limit memory consumption.
gotcha ModuleNotFoundError: Users frequently encounter `ModuleNotFoundError` if `llama-index-llms-ollama` is not installed in the currently active Python environment, or if their IDE (e.g., VS Code's Pylance) is configured to use a different interpreter.
Fix: Ensure the package is installed in your active environment (`pip install llama-index-llms-ollama`) and that your IDE is configured to use the correct Python interpreter. Restarting the IDE can also help.
gotcha Conflicting Ollama Client Versions: There have been reports of conflicts when trying to install both `llama-index-multi-modal-llms-ollama` and `llama-index-llms-ollama` due to differing `ollama-client` version requirements.
Fix: If both integrations are needed, carefully check their respective `ollama-client` dependencies and try to find compatible versions. In some cases, separate virtual environments might be required or reporting the issue to LlamaIndex.

Install

pip install llama-index-llms-ollama Install package

Imports

Ollama
```
from llama_index.llms.ollama import Ollama
```
This is the standard import path for the Ollama LLM integration. Older versions of LlamaIndex might have different structures, but for recent versions, this is correct.

Quickstart

This quickstart demonstrates how to initialize the `Ollama` LLM and use it for both text completion and chat interactions within LlamaIndex. It assumes the Ollama server is running locally and that a model like `llama3.1` has been pulled using `ollama pull llama3.1`. The `request_timeout` is increased for robustness, and `context_window` is an optional parameter for memory management.

# First, ensure Ollama is installed and running, and pull a model:
# On your terminal:
# curl -fsSL https://ollama.com/install.sh | sh
# ollama serve
# ollama pull llama3.1

from llama_index.llms.ollama import Ollama
from llama_index.core.llms import ChatMessage
import os

# Initialize Ollama LLM. Adjust model and timeout as needed.
# Ensure the model 'llama3.1' is pulled via 'ollama pull llama3.1'
llm = Ollama(
    model="llama3.1:latest",
    request_timeout=120.0, # Increase timeout from default 30s if model is slow
    # context_window=8000 # Optionally set context window to limit memory usage
)

# Generate a completion
response_completion = llm.complete("Tell me a short story about a brave knight.")
print("\n--- Completion Response ---")
print(response_completion)

# Send a chat message
messages = [
    ChatMessage(role="system", content="You are a helpful assistant."),
    ChatMessage(role="user", content="What is the capital of France?")
]
response_chat = llm.chat(messages)
print("\n--- Chat Response ---")
print(response_chat.message.content)

view raw JSON →