Brickflows

1.7.0 · active · verified Sun Apr 12

Brickflows is a Python library and CLI tool designed to simplify the development and deployment of scalable workflows on Databricks. It enables users to define Databricks workflows declaratively using Python, leveraging decorators for tasks and integrating seamlessly with Databricks Asset Bundles (DABs) for deployment. The current version is 1.7.0, and it maintains an active release cadence with frequent updates and bug fixes.

Warnings

breaking Deployment failure if 'health' rules were not set (regression in some v1.4.x releases).
Fix: Upgrade to v1.4.1 or later. The regression introduced by #237 was fixed in #245 (v1.4.1), which allows workflows to deploy without explicit health rules.
gotcha The `WorkflowDependencySensor` might not fail when an invalid `dependency_job_id` is provided, leading to silent failures or unexpected behavior.
Fix: Upgrade to v1.3.2 or later, which includes a fix for this behavior. Always thoroughly test dependency sensors.
gotcha Default task settings like `timeout_seconds` might not apply correctly to 'If/Else' or 'For Each' task types in older versions.
Fix: Upgrade to v1.3.3 or later to ensure default task settings are applied consistently across all task types.
gotcha Running a single task directly via programmatic methods in Databricks workflows is not straightforward due to architectural differences from tools like Airflow. Brickflows provides a specific UI-based mechanism.
Fix: To run specific tasks for debugging or testing, use the Databricks UI's 'Run now with different parameters' option and set `brickflow_internal_only_run_tasks` with a comma-separated list of task names. Avoid modifying other parameters.
gotcha Project setup is crucial for correct module resolution and deployment. Incorrectly configured project roots or workflow directories can lead to import errors or deployment issues.
Fix: Always initialize your project using `brickflow projects add` or `bf init` and follow the recommended project structure. Ensure `path_from_repo_root_to_project_root` and `path_project_root_to_workflows_dir` in `.brickflow-project-root.yml` are correctly set.

Install

pip install brickflows Install Brickflows
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sudo sh databricks configure --token Install and Configure Databricks CLI (Prerequisite)

Imports

Workflow
```
from brickflow import Workflow
```
TaskSettings
```
from brickflow import TaskSettings
```
Cluster
```
from brickflow import Cluster
```
Project
```
from brickflow import Project
```
ctx
```
from brickflow.context import ctx
```
Used for accessing Databricks utilities and secrets within tasks.

Quickstart

This quickstart demonstrates how to define a simple 'Hello World' Databricks workflow using Brickflows. It includes defining a workflow with a default cluster and a single task. To deploy, you must first set up a Brickflow project using the `brickflow` CLI and ensure the Databricks CLI is configured to authenticate with your workspace.

from datetime import timedelta
from brickflow import Workflow, Cluster, TaskSettings
import os

# Configure Databricks host and token via environment variables or databricks configure --token
# For this example, ensure your ~/.databrickscfg is set up or environment variables are available.
# Example: export DATABRICKS_HOST="https://<your-workspace-url>.cloud.databricks.com" 
#          export DATABRICKS_TOKEN="dapi..."

wf = Workflow(
    "hello_world_workflow",
    default_cluster=Cluster(
        name="brickflow-example-cluster",
        spark_version='12.2.x-scala2.12',
        node_type_id='Standard_DS3_v2',
        num_workers=1
    ),
    default_task_settings=TaskSettings(
        timeout_seconds=timedelta(hours=2).seconds
    )
)

@wf.task()
def hello_task():
    print(f"Hello from Databricks! Host: {os.environ.get('DATABRICKS_HOST', 'N/A')}")
    return "Task completed successfully"

# To run this, you would typically use the brickflows CLI:
# 1. Create a project: `mkdir my_brickflow_project && cd my_brickflow_project && brickflow projects add`
# 2. Place this code in `workflows/hello_world_wf.py` (assuming workflows is your workflows directory).
# 3. Deploy: `brickflow deploy --deploy-mode=bundle -p <your_databricks_profile>`

view raw JSON →