Databricks SDK for Python

0.102.0 · active · verified Sat Mar 28

The Databricks SDK for Python (Beta) provides a comprehensive client for interacting with the Databricks Lakehouse. It covers all public Databricks REST API operations, offering a robust internal HTTP client that handles intelligent retries. While in Beta, it is supported for production use cases, though future releases are expected to introduce some interface changes. The library is actively developed with frequent releases.

Warnings

breaking The SDK is in Beta, and future releases are expected to introduce interface changes. Databricks recommends pinning dependencies to specific minor versions (e.g., `databricks-sdk==0.102.*`) to avoid unexpected breaking changes during upgrades.
Fix: Pin the minor version of the `databricks-sdk` package in your `requirements.txt` or `pyproject.toml`. Regularly review the changelog before upgrading.
gotcha Authentication errors, often presenting as 'Error: Unable to parse response', typically indicate issues with Databricks host configuration, insufficient permissions for the API operation, or network/firewall problems (e.g., private link redirecting to a login page).
Fix: Verify that `DATABRICKS_HOST` and `DATABRICKS_TOKEN` (or other authentication methods) are correctly configured, and that the principal has the necessary permissions for the API call. Check network connectivity to your Databricks workspace.
gotcha When used in Databricks notebooks, `from databricks.sdk.runtime import *` can lead to namespace pollution, potentially overwriting important objects like the `spark` (SparkSession) object.
Fix: Avoid wildcard imports (`*`). Instead, import specific symbols explicitly, e.g., `from databricks.sdk.runtime import some_function`.
gotcha Within Databricks notebooks, the SparkSession object is automatically initialized by Databricks Runtime. Explicitly initializing a `SparkSession` (e.g., `SparkSession.builder.getOrCreate()`) is redundant and generally unnecessary.
Fix: In Databricks notebooks, simply use the pre-existing `spark` object for Spark operations without explicit initialization.

Install

pip install databricks-sdk Install stable version

Imports

WorkspaceClient
```
from databricks.sdk import WorkspaceClient
```
Main client for workspace-level APIs (e.g., clusters, jobs, notebooks).
AccountClient
```
from databricks.sdk import AccountClient
```
Client for account-level APIs (e.g., users, groups, billing).
Service-specific classes
```
from databricks.sdk.service.compute import ClusterInfo
```
Data classes and enums for specific API services are organized under `databricks.sdk.service`.

Quickstart

This quickstart initializes a `WorkspaceClient` and lists all clusters in the configured Databricks workspace. It leverages the unified authentication mechanism, which automatically detects credentials from environment variables (`DATABRICKS_HOST`, `DATABRICKS_TOKEN`) or a `.databrickscfg` file.

import os
from databricks.sdk import WorkspaceClient

# Databricks SDK uses Databricks unified authentication.
# It prioritizes environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN)
# or a .databrickscfg file. For this example to run outside Databricks,
# ensure these are set.
# Example: export DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
# Example: export DATABRICKS_TOKEN=dapi********************************

host = os.environ.get('DATABRICKS_HOST', 'https://your-workspace.cloud.databricks.com')
token = os.environ.get('DATABRICKS_TOKEN', 'dapi_your_token_here')

try:
    # Initialize WorkspaceClient, which will pick up credentials automatically.
    # For explicit config:
    # w = WorkspaceClient(host=host, token=token)
    w = WorkspaceClient()

    print(f"Listing clusters in Databricks workspace: {w.config.host}")
    for c in w.clusters.list():
        print(f"  - {c.cluster_name} (ID: {c.cluster_id})")

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure DATABRICKS_HOST and DATABRICKS_TOKEN environment variables are set ")
    print("or a valid .databrickscfg file exists with proper authentication.")

view raw JSON →