Dagster Azure

Warnings

• gotcha The `dagster-azure` library's version (`0.x.x`) is aligned with the core `dagster` version (`1.x.x`). For example, `dagster-azure==0.29.0` is compatible with `dagster==1.13.0`. Always ensure that your `dagster` and `dagster-azure` versions are compatible, typically by installing them together (e.g., `pip install dagster==1.13.0 dagster-azure==0.29.0`).
  Fix: Refer to the official Dagster release notes for version compatibility between core Dagster and its libraries. Always upgrade both core Dagster and its libraries together.
• gotcha Azure authentication relies on `azure-identity`'s `DefaultAzureCredential`. This credential provider attempts various authentication methods (environment variables, managed identity, Azure CLI, etc.). Incorrectly configured authentication is a common source of errors.
  Fix: Ensure the necessary environment variables (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`) are set, or that your execution environment has a configured Managed Identity or Azure CLI login. Consult Azure identity documentation for `DefaultAzureCredential` setup.
• breaking Major version updates to core `dagster` (e.g., `1.x.x` to `2.x.x` when it eventually happens) or significant changes within `1.x.x` can introduce breaking changes to resource definitions, I/O manager interfaces, or configuration schema that `dagster-azure` components rely on. This might require updates to your resource and I/O manager definitions.
  Fix: Always review the release notes for `dagster` and `dagster-azure` when upgrading. Pay close attention to changes in resource configuration, I/O manager behavior, and any deprecation warnings in the console or UI.

Install

• pip install dagster-azure Install dagster-azure

Imports

• azure_blob_storage_resource

  from dagster_azure.blob.resource import azure_blob_storage_resource

• blob_storage_io_manager

  from dagster_azure.blob.io_manager import blob_storage_io_manager

• adls2_file_cache

  from dagster_azure.adls2 import adls2_file_cache

• AzureDataLakeGen2Resource

  from dagster_azure.adls2.resources import AzureDataLakeGen2Resource

Quickstart

This example defines an asset `hello_blob_asset` that returns a string. It is configured to use the `blob_storage_io_manager` to persist this string to an Azure Blob Storage container. The `io_manager` automatically handles serialization and deserialization. Ensure you have the `AZURE_STORAGE_ACCOUNT_NAME` and `AZURE_BLOB_CONTAINER_NAME` environment variables set, and that your environment is configured for Azure authentication (e.g., via `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`). To run this example, save it as `repo.py` and run `dagster dev` in the same directory, then launch the `hello_blob_job` from the UI.

import os
from dagster import Definitions, asset, JobDefinition
from dagster_azure.blob.io_manager import blob_storage_io_manager

# Set these environment variables or replace with actual values
# For authentication, 'DefaultAzureCredential' (used by dagster-azure) looks for:
# AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET or AZURE_FEDERATED_TOKEN_FILE
# or uses Managed Identity.
AZURE_STORAGE_ACCOUNT_NAME = os.environ.get(
    "AZURE_STORAGE_ACCOUNT_NAME", "your_storage_account_name"
)
AZURE_BLOB_CONTAINER_NAME = os.environ.get(
    "AZURE_BLOB_CONTAINER_NAME", "your-dagster-container"
)

@asset
def hello_blob_asset():
    """An asset that writes a simple string to Azure Blob Storage."""
    return "Hello, Dagster Azure Blob Storage!"

# Create a job that materializes the asset
hello_blob_job = JobDefinition(name="hello_blob_job", assets=[hello_blob_asset])

defs = Definitions(
    assets=[hello_blob_asset],
    jobs=[hello_blob_job],
    resources={
        "io_manager": blob_storage_io_manager.configured({
            "storage_account_name": AZURE_STORAGE_ACCOUNT_NAME,
            "container": AZURE_BLOB_CONTAINER_NAME,
            "prefix": "dagster_output/" # Optional: objects will be stored under this prefix
        })
    }
)