DataHub Python CLI and SDK

1.5.0.5 · active · verified Thu Apr 09

The `acryl-datahub` package provides a powerful Command Line Interface (CLI) and a Python SDK for interacting with DataHub, an open-source metadata platform. DataHub serves as a central nervous system for your data stack, enabling discovery, governance, and observability across various data assets. Currently at version 1.5.0.5, the library maintains an active release cadence with frequent updates and release candidates, ensuring ongoing feature development and stability.

Warnings

breaking Python 3.9 support has been officially dropped. All `acryl-datahub` packages now require Python 3.10 or later.
Fix: Upgrade your Python environment to version 3.10 or newer before upgrading `acryl-datahub`.
breaking The V1 UI theme is officially sunset as of v1.5.0. All development targets the V2 UI going forward. If you're self-hosting, ensure your GMS environment variables `THEME_V2_ENABLED` and `THEME_V2_DEFAULT` are set to `true`.
Fix: Set `THEME_V2_ENABLED=true` and `THEME_V2_DEFAULT=true` in your DataHub GMS configuration. The `THEME_V2_TOGGLEABLE` variable should also be set to `false`.
breaking The `acryl-datahub` package now requires Pydantic v2. Support for Pydantic v1 has been dropped.
Fix: Ensure `pydantic>=2.0` is installed in your environment. If you have other packages requiring Pydantic v1, consider using separate virtual environments.
breaking SQL view query IDs now use SHA-256 hashes instead of URL-encoding the view URN. This means old query entities for view lineage tracking will become orphaned.
Fix: Use stateful ingestion to clean up and re-ingest view lineage to generate new URNs based on the SHA-256 hash.
gotcha For DataHub CLI version 1.5, the handling of the token signing key for Metadata Service Authentication has changed. If not explicitly set via environment variables, new random values are generated and stored locally (`~/.datahub/quickstart/.local-secrets.env`).
Fix: For production deployments, explicitly set `DATAHUB_TOKEN_SERVICE_SIGNING_KEY` and `DATAHUB_TOKEN_SERVICE_SALT` environment variables to your own secure values.
gotcha The `DatahubRestEmitter.emit()` method (and `emit_mcp()`) now returns `Optional[TraceData]` instead of `None` or an `int`. This change exposes trace IDs for SYNC_PRIMARY and ASYNC modes.
Fix: Update any code that expects a `None` or `int` return type from `emit()` or `emit_mcp()`. The return value should now be checked for `TraceData` if trace information is needed.

Install

pip install acryl-datahub Install core package
pip install 'acryl-datahub[datahub-rest]' # For programmatic interaction over REST Install with REST emitter

Imports

DatahubRestEmitter
```
from datahub.emitter.rest_emitter import DatahubRestEmitter
```
Used for sending metadata changes to DataHub over REST.
MetadataChangeProposalWrapper
```
from datahub.emitter.mcp import MetadataChangeProposalWrapper
```
A wrapper for constructing metadata change proposals.
DatasetPropertiesClass
```
from datahub.metadata.schema_classes import DatasetPropertiesClass
```
Example of a generated schema class for common metadata aspects.
DatahubClientConfig
```
from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph
```
Used for configuring and interacting with the DataHub GraphQL API programmatically.

Quickstart

This quickstart first outlines how to set up a local DataHub instance using the CLI's `docker quickstart` command. Following this, it provides a Python snippet demonstrating how to programmatically connect to a DataHub server using the `DatahubRestEmitter` and publish basic dataset properties.

import os
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import DatasetPropertiesClass

# --- CLI Quickstart (run in your terminal) ---
# 1. Install Docker and Docker Compose v2.
# 2. Start a local DataHub instance:
#    datahub docker quickstart
#    (This command might take some time to download and start services)
#
# --- Python SDK Example (after DataHub is running) ---
# For local quickstart, GMS server is typically http://localhost:8080
gms_server = os.environ.get("DATAHUB_GMS_SERVER", "http://localhost:8080")
token = os.environ.get("DATAHUB_GMS_TOKEN", "") # For cloud/secured instances, provide a token

# Initialize the REST emitter
# Note: The 'token' parameter is available for direct use, not just extra_headers.
emitter = DatahubRestEmitter(gms_server=gms_server, token=token)

# Define a sample dataset URN
dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:hive,sample_dataset,PROD)"

# Create a DatasetProperties aspect
dataset_properties = DatasetPropertiesClass(
    description="This is a sample dataset emitted via the Python SDK quickstart.",
    customProperties={
        "owner_team": "data_platform",
        "environment": "production_dev"
    }
)

# Create a MetadataChangeProposalWrapper
mcp = MetadataChangeProposalWrapper(
    entityUrn=dataset_urn,
    aspect=dataset_properties,
)

# Emit the metadata change proposal
try:
    emitter.emit(mcp)
    print(f"Successfully emitted properties for dataset: {dataset_urn}")
except Exception as e:
    print(f"Failed to emit metadata: {e}")
    print("Ensure your DataHub instance is running and accessible at", gms_server)

view raw JSON →