Databricks Bundles (Declarative Automation Bundles)

0.296.0 · active · verified Mon Apr 13

Databricks Bundles, recently renamed to Declarative Automation Bundles, provides Python support for defining, dynamically creating, and modifying Databricks jobs and pipelines. It extends the core Declarative Automation Bundles functionality, allowing users to apply software engineering best practices like source control, code review, testing, and CI/CD to their data and AI projects. The library is currently at version 0.296.0 and is actively maintained, with a focus on streamlining deployments and enabling programmatic configuration through Python and YAML files, orchestrated via the Databricks CLI.

Warnings

breaking The product name changed from 'Databricks Asset Bundles' to 'Declarative Automation Bundles'. While the `bundle` CLI command remains the same, this indicates a conceptual shift and continuous evolution of the platform.
Fix: No breaking changes to existing CLI commands or configurations are expected, but be aware of the new terminology in documentation and communications.
gotcha Directly editing deployed notebooks or jobs in the Databricks workspace UI can lead to configuration drift and unexpected behavior during subsequent bundle deployments. The local bundle repository is considered the source of truth.
Fix: Always make changes in your local bundle project files (Python or YAML) and then `databricks bundle deploy`. For debugging, copy the notebook to a scratch folder outside the bundle path or debug locally.
gotcha Permission denied errors (e.g., `CAN MANAGE`, `USE CATALOG`) are common if the service principal or user deploying the bundle lacks the necessary permissions on jobs, Unity Catalog, or other resources.
Fix: Explicitly grant all required permissions to the deploying identity (Service Principal for CI/CD, human user for local development). Ensure Unity Catalog objects (catalogs, schemas) exist and have appropriate access controls.
breaking Workspace paths in bundle configurations are now automatically prefixed with `/Workspace` (Databricks CLI 0.230.0+). Using path strings like `/Workspace/${workspace.root_path}/...` will generate a warning and be replaced.
Fix: Update bundle configurations to remove explicit `/Workspace` prefixes. Paths should start with `${workspace.root_path}/...` or relative paths within the bundle structure. Validate your bundle to catch warnings.
breaking The fallback path resolution behavior for resources defined in one file and overridden in another was removed in Databricks CLI 0.266.0. This could lead to confusing and error-prone path resolution in older configurations.
Fix: Review and update bundle configurations that rely on implicit fallback paths, ensuring explicit and unambiguous path definitions for all resources and overrides.
gotcha The default 'dev' target created by `databricks bundle init` might not automatically align with your development Git branch or expected development environment. This requires careful manual configuration.
Fix: Clearly define your target environments in `databricks.yml`, mapping them explicitly to your desired Databricks workspaces and configuration settings. Avoid assumptions about the 'dev' target's behavior.

Install

pip install databricks-bundles Install Python package (for Python-defined bundles)
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh databricks -v # Verify installation databricks auth login --host https://<your-workspace-url> Install and Configure Databricks CLI (required)

Imports

Not applicable for direct application-level imports
```
The 'databricks-bundles' Python package is primarily used by the Databricks CLI internally when processing Python-defined bundle resources, rather than direct 'from pkg import ClassName' statements in end-user application code.
```
User code defines resources (e.g., jobs, pipelines) in Python files that the CLI then interprets, rather than importing and calling classes/functions from `databricks-bundles` directly within a running Python application.

Quickstart

To get started with Databricks Bundles, first ensure the Databricks CLI is installed and authenticated. Then, initialize a new bundle project using a template, typically `experimental-jobs-as-code` for Python support. Define your Databricks resources (like jobs or pipelines) in `databricks.yml` or dedicated Python files within the bundle structure. Finally, use the CLI commands `databricks bundle validate`, `databricks bundle deploy`, and `databricks bundle run` to manage and execute your project on Databricks.

# 1. Initialize a new bundle project (select 'Default Python' template when prompted)
databricks bundle init --template experimental-jobs-as-code

# 2. Navigate into the new project directory
cd <your-bundle-project-name>

# 3. Create a Python file for a job task (e.g., src/my_job.py)
#    Content for src/my_job.py:
#    print("Hello from my Databricks Bundle!")

# 4. Define a simple job in databricks.yml (or a Python resource definition if using Python bundles)
#    Example databricks.yml snippet defining a job running my_job.py (ensure 'target: dev' matches your config)
#    bundle:
#      name: my-first-bundle
#    resources:
#      jobs:
#        my_example_job:
#          name: MyExampleJob
#          tasks:
#            - task_key: run_script
#              python_file_task:
#                python_file: src/my_job.py
#              new_cluster:
#                spark_version: 13.3.x-scala2.12
#                node_type_id: Standard_DS3_v2
#                num_workers: 1
#    targets:
#      dev:
#        default: true
#        workspace:
#          host: https://<your-workspace-url>

# 5. Validate the bundle configuration
databricks bundle validate

# 6. Deploy the bundle to your Databricks workspace
databricks bundle deploy --target dev

# 7. Run the deployed job
databricks bundle run my_example_job --target dev

view raw JSON →