Databricks PyPI Extras

0.1 · active · verified Thu Apr 16

`databricks-pypi-extras` is a Python library developed by Databricks, providing a collection of utilities designed to enhance the Databricks user experience and extend existing PyPI libraries for better compatibility with Databricks environments. It currently includes modules like `databricks.connect_extras` to simplify operations such as connecting to Databricks (e.g., via Databricks Connect v2) and interacting with notebook contexts. The current version is 0.1, with releases likely occurring as new utilities are developed and integrated.

Common errors

```
ModuleNotFoundError: No module named 'databricks.connect_extras.context'
```
cause The `databricks-pypi-extras` library or a specific submodule was not installed, or the import path is incorrect.

fix Ensure you have installed the library using `pip install databricks-pypi-extras`. Verify the import path against the official GitHub repository's `src` directory, as the library is modular and may introduce new sub-packages.
```
Spark session could not be retrieved. Ensure Databricks Connect is properly configured or run in a Databricks Notebook.
```
cause Attempting to use `current_spark_context()` or similar utilities from `databricks.connect_extras` outside of a properly configured Databricks Connect environment or a Databricks Notebook.

fix Install `databricks-connect` (`pip install "databricks-connect[databricks-connect-dependencies]"`) and configure it using `databricks-connect configure`, or run your code directly within a Databricks notebook.

Warnings

gotcha Many utilities in `databricks-pypi-extras` (especially in `connect_extras`) are designed for specific Databricks environments (e.g., Databricks Notebooks or Databricks Connect v2). Running them outside these environments might lead to `None` returns, errors, or unexpected behavior.
Fix: Ensure your code is running within a Databricks Notebook or that your local environment is correctly configured with `databricks-connect` if using `connect_extras` features.
breaking As the library is currently at version 0.1, its API is considered experimental and highly subject to change. Future versions may introduce breaking changes to existing modules, functions, or their signatures without prior deprecation cycles.
Fix: Pin your dependency to `databricks-pypi-extras==0.1` and monitor the GitHub repository's `main` branch or future releases for API updates before upgrading to newer versions. Be prepared to adapt your code when upgrading.

Install

pip install databricks-pypi-extras Install core library

Imports

current_spark_context
```
from databricks.connect_extras.context import current_spark_context
```
A utility to get the active SparkSession, typically from Databricks Connect or a Databricks notebook environment.
get_current_notebook_path
```
from databricks.connect_extras.notebook import get_current_notebook_path
```
A utility to retrieve the path of the current Databricks notebook.

Quickstart

This quickstart demonstrates how to use `current_spark_context` from `databricks.connect_extras` to obtain a SparkSession. This utility is particularly useful when working with Databricks Connect v2 or directly within a Databricks notebook environment. Outside these contexts, it requires `databricks-connect` to be installed and configured.

import os
from databricks.connect_extras.context import current_spark_context

# This example demonstrates retrieving a Spark session using databricks.connect_extras.
# To run this successfully outside a Databricks Notebook, you must:
# 1. Install Databricks Connect: pip install "databricks-connect[databricks-connect-dependencies]"
# 2. Configure Databricks Connect using `databricks-connect configure`
#    or by setting environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_CLUSTER_ID, etc.).

print("Attempting to get Spark session via databricks.connect_extras...")
try:
    spark = current_spark_context()
    if spark:
        print(f"Successfully retrieved SparkSession (Spark version: {spark.version})")
        # Example usage: create a simple DataFrame
        data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
        df = spark.createDataFrame(data, ["Name", "Value"])
        print("\nExample DataFrame created and shown:")
        df.show()
    else:
        print("Spark session could not be retrieved. Ensure Databricks Connect is properly configured or run in a Databricks Notebook.")
except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure `databricks-connect` is installed and configured, and your environment is set up for Databricks Connect v2.")

view raw JSON →