Delta Sharing Python Connector

Warnings

• gotcha Linux users may encounter installation issues for `delta-kernel-rust-sharing-wrapper` if `glibc` version is older than 2.31 or if a pre-built Python wheel is not available for their environment.
  Fix: Ensure your Linux system has `glibc >= 2.31`. If installation still fails, install the Rust toolchain (e.g., via `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`) as `pip` might try to build the Rust component from source.
• gotcha Delta Sharing profile files (`.share`) contain sensitive credentials (e.g., bearer tokens, OAuth client secrets). These files must be stored securely and not exposed in public repositories or insecure locations.
  Fix: Store profile files in secure, access-controlled locations (e.g., local filesystem with restricted permissions, cloud storage with IAM policies, or secret management services). Avoid hardcoding credentials directly in code. For OIDC authentication, ensure `clientId` and `clientSecret` are managed securely.
• gotcha When using `load_as_spark()` to read shared tables as Spark DataFrames, you must be running in a PySpark environment with the Apache Spark Connector for Delta Sharing properly configured and installed.
  Fix: Refer to the official Delta Sharing documentation for setting up the Apache Spark Connector for Delta Sharing in your Spark environment, including necessary Spark package configurations (e.g., `--packages io.delta:delta-sharing-spark_2.12:<version>`).
• gotcha Bearer tokens used for open sharing have a maximum validity of one year. Recipients must coordinate with data providers for token rotation and renewal to maintain access.
  Fix: Implement a process for regularly renewing and rotating bearer tokens with your data provider well before their expiration. For OIDC federation, token management is typically handled more dynamically.

Install

• pip install delta-sharing Base Installation
• pip install delta-sharing[s3] With S3 Cloud Storage Support (example)

Imports

• SharingClient

  from delta_sharing import SharingClient

  The primary client for interacting with Delta Sharing servers.
• load_as_pandas

  from delta_sharing import load_as_pandas

  Function to load a shared table directly into a pandas DataFrame.
• load_as_spark

  from delta_sharing import load_as_spark

  Function to load a shared table directly into a PySpark DataFrame (requires PySpark environment).
• list_all_tables

  client.list_all_tables()

  Method of SharingClient to list all available tables.

Quickstart

This quickstart demonstrates how to initialize the Delta Sharing client, list available shared tables, and load a sample table into a pandas DataFrame. It assumes you have a Delta Sharing profile file (e.g., `open-datasets.share`) that provides credentials to a Delta Sharing server. For demonstration, it attempts to load a publicly available dataset.

import delta_sharing
import os

# Point to a Delta Sharing profile file (e.g., downloaded from a data provider)
# For a public example, you can use:
# profile_file = "https://raw.githubusercontent.com/delta-io/delta-sharing/main/examples/open-datasets.share"
# In a real scenario, this would be a local path or cloud storage path (e.g., s3://bucket/profile.share)
# Ensure your profile file (e.g., 'config.share') is accessible.
# For local testing, download from https://databricks-datasets-oregon.s3-us-west-2.amazonaws.com/delta-sharing/share/open-datasets.share
# and save it as 'open-datasets.share' in your working directory.

profile_file = os.environ.get('DELTA_SHARING_PROFILE', 'open-datasets.share')

try:
    # Create a SharingClient
    client = delta_sharing.SharingClient(profile_file)

    # List all shared tables
    print("\nAvailable Shares, Schemas, and Tables:")
    tables = client.list_all_tables()
    if not tables:
        print("No tables found. Ensure your profile file is correct and has access.")
    for table in tables:
        print(f"  - Share: {table.share}, Schema: {table.schema}, Table: {table.name}")

    # Example: Load a specific table (replace with a table from your profile if needed)
    # Using the 'COVID_19_NYT' table from the open-datasets.share example
    # The format is <profile-path>#<share>.<schema>.<table>
    example_table_url = f"{profile_file}#delta_sharing.default.COVID_19_NYT"
    print(f"\nLoading data from: {example_table_url}")
    
    # Load the table as a pandas DataFrame, with a limit for demonstration
    df = delta_sharing.load_as_pandas(example_table_url, limit=5)
    print("\nFirst 5 rows of the DataFrame:")
    print(df)

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure you have a valid Delta Sharing profile file configured and accessible.")
    print("You can set the DELTA_SHARING_PROFILE environment variable or download 'open-datasets.share'.")