Delta Lake Python APIs for Apache Spark

4.1.0 · active · verified Sat Mar 28

delta-spark provides Python APIs to interact with Delta Lake tables using Apache Spark. It enables operations like reading, writing, and time-traveling Delta tables, leveraging Spark's distributed processing capabilities. The library maintains a rapid release cadence, often releasing multiple patch and minor versions for each major iteration. The current version is 4.1.0.

Warnings

breaking The preview feature for catalog-managed tables was renamed from `catalogOwned-preview` to `catalogManaged` in v4.0.1. Legacy `ucTableId` also transitioned to `io.unitycatalog.tableId`.
Fix: If using catalog-managed tables in v4.0.0, update Spark configurations and any code referencing the feature to use `catalogManaged` and `io.unitycatalog.tableId` when upgrading to v4.0.1 or later.
gotcha Each `delta-spark` release is built against and optimized for specific Apache Spark versions. While some backward compatibility exists (e.g., Delta 4.1.0 supports Spark 4.1.0 and 4.0.1), major Spark version upgrades can introduce incompatibilities or require specific `delta-spark` versions.
Fix: Always consult the official Delta Lake release notes and documentation to ensure your `delta-spark` and `pyspark` versions are compatible. Upgrade both in tandem if necessary.
gotcha The 'catalog-managed tables' feature introduced in v4.0.0 (preview) was explicitly stated to be in an RFC stage and 'subject to change'. Early adopters of this feature in v4.0.0 experienced breaking changes in v4.0.1.
Fix: For production systems, exercise caution with 'preview' features. If using such features, be prepared for API changes and thoroughly test upgrades. Migrate to the stable naming conventions in later versions.
gotcha Starting with version 4.x, `delta-spark` requires Python 3.10 or newer.
Fix: Ensure your Python environment is 3.10 or higher. You can check your Python version using `python --version`.

Install

pip install delta-spark Install delta-spark

Imports

DeltaTable
```
from delta.tables import DeltaTable
```
Used for DML operations (update, delete, merge) and managing Delta tables.

SparkSession config for Delta

SparkSession.builder.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension").config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

Essential configuration to enable Delta Lake features within a SparkSession.

Quickstart

This quickstart demonstrates how to initialize a SparkSession with Delta Lake extensions, write a DataFrame to a Delta table, and then read the data back. It also shows how to get details about the Delta table using the `DeltaTable` API.

from pyspark.sql import SparkSession
from delta.tables import DeltaTable
import os

# Configure SparkSession for Delta Lake
spark = (
    SparkSession.builder.appName("DeltaSparkQuickstart")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config(
        "spark.sql.catalog.spark_catalog",
        "org.apache.spark.sql.delta.catalog.DeltaCatalog",
    )
    .getOrCreate()
)

# Create a simple DataFrame
data = spark.createDataFrame([(1, "Alice"), (2, "Bob")], ["id", "name"])

# Define a path for the Delta table
delta_table_path = os.path.join(os.getcwd(), "tmp", "delta_table")

# Write data to a Delta table
print(f"Writing data to Delta table at: {delta_table_path}")
data.write.format("delta").mode("overwrite").save(delta_table_path)

# Read data from the Delta table
print(f"Reading data from Delta table at: {delta_table_path}")
df_read = spark.read.format("delta").load(delta_table_path)
df_read.show()

# Use DeltaTable API for operations (e.g., detail)
delta_table = DeltaTable.forPath(spark, delta_table_path)
print("Delta table description:")
delta_table.detail().show()

# Stop SparkSession
spark.stop()

view raw JSON →