Foundry Local SDK

1.0.0 · active · verified Thu Apr 16

Foundry Local SDK for Python provides a control-plane interface to Foundry Local, a unified on-device AI runtime that enables local generative AI inference. It allows developers to interact with locally run models, perform chat completions, and manage local AI resources without network latency or per-token costs. The current version is 1.0.0, and it has moved to General Availability, suggesting a stable, but potentially evolving, release cadence.

Common errors

```
ConnectionRefusedError: [Errno 111] Connection refused
```
cause The Foundry Local runtime service is not running or is not accessible at its default port (usually 8008 or similar). The Python SDK cannot connect to the local AI engine.

fix Start the Foundry Local runtime application/service. Verify its status using the `foundry status` CLI command. Ensure no firewall is blocking access to its local port.
```
foundry.exceptions.ModelNotFoundError: Model 'my-non-existent-model' not found
```
cause You are trying to use a model name that has not been downloaded to your local Foundry Local instance, or the model name is misspelled.

fix Download the required model using the Foundry Local CLI (`foundry download <model_name>`) or `client.download_model(<model_name>)` via the SDK. Verify the exact model name from `foundry list`.
```
ImportError: cannot import name 'FoundryLocal' from 'foundry_local_sdk'
```
cause Incorrect import statement. The top-level package for `foundry-local-sdk` exposes the main class via `foundry` not `foundry_local_sdk`.

fix Change your import statement to `from foundry import FoundryLocal`.
```
ValueError: max_tokens must be positive
```
cause The `max_tokens` parameter for `chat.completions.create` was provided with a non-positive value (e.g., 0 or a negative number).

fix Ensure `max_tokens` is set to a positive integer value greater than 0, corresponding to the maximum number of tokens you want the model to generate.

Warnings

gotcha The `foundry-local-sdk` Python library is a client to the Foundry Local runtime. It requires the Foundry Local application/service to be installed and running on your system for any API calls to function.
Fix: Install the Foundry Local runtime from official Microsoft sources and ensure it's running before using the Python SDK. Check its status via the `foundry` CLI.
gotcha Models used with the SDK must be downloaded to your local Foundry Local instance. Attempting to use a model that hasn't been downloaded will result in a `ModelNotFoundError`.
Fix: Use the `foundry download <model_name>` CLI command or `client.download_model(<model_name>)` in the SDK to fetch the desired model before attempting inference.
gotcha Early versions (pre-1.0.0, e.g., v0.8.113) had a limitation of supporting only one tool call per request. While this might be improved in newer versions, be aware of potential limitations with complex tool-calling scenarios.
Fix: Refer to the latest Foundry Local documentation for current tool-calling capabilities and limitations. For older versions, simplify tool_choice to ensure only one tool is invoked.
gotcha Some users experienced issues with tool calling on NVIDIA GPUs with older Foundry Local runtime versions (e.g., v0.8.117). This indicates potential hardware-specific runtime bugs.
Fix: Ensure your Foundry Local runtime and SDK are updated to the latest available version, as many such issues are resolved in subsequent releases. Check official release notes for hardware-specific fixes.

Install

pip install foundry-local-sdk Install stable version

Imports

FoundryLocal
wrong:
```
from foundry_local_sdk import FoundryLocal
```
correct:
```
from foundry import FoundryLocal
```
The package installs as 'foundry-local-sdk', but the primary module is `foundry`, from which `FoundryLocal` is imported.

Quickstart

Initializes the Foundry Local client and performs a simple chat completion using a locally available model. Before running, ensure the Foundry Local runtime is installed and running, and a model (e.g., `phi3-mini-4k-instruct`) has been downloaded via the `foundry` CLI.

from foundry import FoundryLocal
import os

# Ensure the Foundry Local runtime is installed and running.
# For example, download the 'phi3-mini-4k-instruct' model via the Foundry Local CLI:
# foundry download phi3-mini-4k-instruct

try:
    client = FoundryLocal()
    # You might need to explicitly download a model if not already present.
    # client.download_model("phi3-mini-4k-instruct") # Uncomment and run once if needed

    print("FoundryLocal client initialized. Attempting chat completion...")
    response = client.chat.completions.create(
        model="phi3-mini-4k-instruct", # Replace with a model you have downloaded
        messages=[
            {"role": "user", "content": "What is the capital of France?"}
        ],
        max_tokens=50
    )
    print("Response:", response.choices[0].message.content)
except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure the Foundry Local runtime is installed, running, and the specified model is downloaded.")

view raw JSON →