Databricks Connect Client

18.1.2 · active · verified Thu Apr 09

Databricks Connect allows you to connect popular IDEs, notebook servers, and custom applications to Databricks clusters. It's a client library that configures standard PySpark APIs to run commands remotely on Databricks clusters, enabling local development and debugging against data on a remote cluster. The current version is 18.1.2, with releases tied closely to Databricks Runtime (DBR) versions, typically aligning with DBR major/LTS releases and subsequent patch updates.

Warnings

breaking Databricks Connect client versions are strictly tied to specific Python versions. For instance, Databricks Connect 18.x requires Python 3.12. Using an incompatible Python version will result in installation failures or runtime errors.
Fix: Ensure your local Python environment's major and minor version exactly matches the requirement for your Databricks Connect client version (e.g., Python 3.12 for databricks-connect==18.x).
breaking The Databricks Connect client version must exactly match the Databricks Runtime (DBR) version of the cluster you are connecting to. Mismatched versions will lead to connection errors or unexpected behavior.
Fix: Check your cluster's DBR version and install the corresponding `databricks-connect` client version (e.g., for DBR 13.3 LTS, install `databricks-connect==13.3.*`).
gotcha Installing `databricks-connect` with an existing `pyspark` installation can lead to dependency conflicts or unexpected `pyspark` version mismatches. `databricks-connect` bundles its own compatible `pyspark`.
Fix: It is highly recommended to install `databricks-connect` in a clean Python virtual environment. Do not install `pyspark` separately when using `databricks-connect`.
gotcha Incorrect configuration of connection parameters (host, token, cluster ID, sometimes org ID or port) is a very common reason for connection failures.
Fix: Double-check that `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, and `DATABRICKS_CLUSTER_ID` environment variables (or values passed directly to SparkSession builder) are correct and correspond to your target Databricks cluster and workspace. Ensure your token has sufficient permissions.

Install

pip install databricks-connect==18.1.2 Install specific version

Imports

SparkSession
```
from pyspark.sql import SparkSession
```
Databricks Connect configures the standard PySpark SparkSession, you don't import SparkSession directly from databricks.connect.

Quickstart

This quickstart demonstrates how to initialize a SparkSession with Databricks Connect using environment variables for configuration. Ensure your `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, and `DATABRICKS_CLUSTER_ID` are set correctly. The `display()` command requires a Databricks environment; use `show()` for local console output.

import os
from pyspark.sql import SparkSession

# Configure environment variables (replace with your actual values)
# For DBR 13.x and later, DATABRICKS_CLUSTER_ID is required.
# For DBR 12.x and earlier, DATABRICKS_ORG_ID might be required.
# DATABRICKS_PORT is optional, defaults to 15001.

os.environ['DATABRICKS_HOST'] = os.environ.get('DATABRICKS_HOST', 'https://your-databricks-instance.cloud.databricks.com')
os.environ['DATABRICKS_TOKEN'] = os.environ.get('DATABRICKS_TOKEN', 'dapi...')
os.environ['DATABRICKS_CLUSTER_ID'] = os.environ.get('DATABRICKS_CLUSTER_ID', 'your-cluster-id')
# os.environ['DATABRICKS_ORG_ID'] = os.environ.get('DATABRICKS_ORG_ID', 'your-org-id') # Often not needed for modern DBR/configurations
# os.environ['DATABRICKS_PORT'] = os.environ.get('DATABRICKS_PORT', '15001') # Default is 15001

# Initialize SparkSession using Databricks Connect
# The .builder.getOrCreate() method automatically picks up DATABRICKS_ environment variables.
spark = SparkSession.builder.getOrCreate()

# Example: Run a simple Spark command
df = spark.range(10).toDF("id")
df.display()
# df.show() # For local console output

print("Successfully connected to Databricks cluster and ran a Spark command.")

# Clean up environment variables if running multiple tests or configurations
# del os.environ['DATABRICKS_HOST']
# del os.environ['DATABRICKS_TOKEN']
# del os.environ['DATABRICKS_CLUSTER_ID']
# (and others you set)

view raw JSON →