{"id":1445,"library":"dbt-databricks","title":"dbt-databricks Adapter","description":"The `dbt-databricks` library is an adapter plugin that allows dbt (data build tool) to connect to and transform data within Databricks environments. It supports Databricks SQL Endpoints and clusters, enabling users to leverage dbt's data transformation capabilities with Delta Lake tables and Unity Catalog. It is currently at version 1.11.6 and its release cycle typically aligns closely with `dbt-core` releases, ensuring compatibility.","status":"active","version":"1.11.6","language":"en","source_language":"en","source_url":"https://github.com/dbt-labs/dbt-databricks","tags":["dbt","databricks","etl","elt","sql","data-warehousing","adapter","unity-catalog"],"install":[{"cmd":"pip install dbt-databricks","lang":"bash","label":"Install dbt-databricks"}],"dependencies":[{"reason":"dbt-databricks is an adapter for dbt-core and depends on it implicitly for functionality. Installing dbt-databricks will typically pull in a compatible dbt-core version.","package":"dbt-core","optional":false}],"imports":[{"note":"This is an internal dbt-core import for adapter registration and is not typically used directly by end-users. `dbt-databricks` functionality is primarily accessed via the dbt CLI and `profiles.yml` configuration.","symbol":"DatabricksAdapter","correct":"from dbt.adapters.databricks import DatabricksAdapter"}],"quickstart":{"code":"import os\nimport sys\n\n# --- Step 1: Set up environment variables for authentication ---\n# In a real scenario, set DBT_DATABRICKS_TOKEN securely,\n# e.g., via your shell or CI/CD secrets.\n# For local testing, replace 'YOUR_DATABRICKS_PAT' with an actual PAT.\n# Using os.environ.get for compliance with auth checks.\ndatabricks_token = os.environ.get('DBT_DATABRICKS_TOKEN', 'YOUR_DATABRICKS_PAT_FOR_QUICKSTART_ONLY')\nif databricks_token == 'YOUR_DATABRICKS_PAT_FOR_QUICKSTART_ONLY':\n    print(\"WARNING: DBT_DATABRICKS_TOKEN not set in environment. Using placeholder. Ensure you replace it.\")\nos.environ['DBT_DATABRICKS_TOKEN'] = databricks_token\n\n# --- Step 2: Verify dbt-databricks installation ---\ntry:\n    import dbt.adapters.databricks # Check if the package is findable\n    print(\"dbt-databricks is installed and accessible.\")\nexcept ImportError:\n    print(\"ERROR: dbt-databricks not found. Please run 'pip install dbt-databricks'\")\n    sys.exit(1)\n\nprint(\"\\n--- Quickstart: Next Steps (run these in your terminal) ---\")\nprint(\"1. Configure your dbt profile in `~/.dbt/profiles.yml`:\")\nprint(\"   Replace placeholders like `<your-databricks-workspace-url>`, etc.\")\nprint(\"   Example `profiles.yml` snippet (named `my_databricks_project`):\")\nprint(\"\\n      my_databricks_project:\")\nprint(\"        target: dev\")\nprint(\"        outputs:\")\nprint(\"          dev:\")\nprint(\"            type: databricks\")\nprint(\"            host: <your-databricks-workspace-url> # e.g., dbc-xxxx.cloud.databricks.com\")\nprint(\"            http_path: /sql/1.0/warehouses/<your-sql-warehouse-id> # For SQL Endpoints\")\nprint(\"            token: \"\"{{ env_var('DBT_DATABRICKS_TOKEN') }}\"\" # Uses the env var set above\")\nprint(\"            catalog: <your-unity-catalog-name> # Optional, if using Unity Catalog\")\nprint(\"            schema: dbt_quickstart_schema\")\nprint(\"            threads: 4\")\nprint(\"\\n2. Initialize a new dbt project and link your profile:\")\nprint(\"   mkdir my_dbt_project_databricks && cd my_dbt_project_databricks\")\nprint(\"   dbt init --skip-profile-setup\")\nprint(\"   # Edit `dbt_project.yml` to set `profile: 'my_databricks_project'`\")\nprint(\"\\n3. Test your Databricks connection:\")\nprint(\"   dbt debug --profile my_databricks_project\")\nprint(\"\\n4. Run your dbt models (after creating some in the `models` directory):\")\nprint(\"   dbt run --profile my_databricks_project\")","lang":"python","description":"This quickstart demonstrates setting up environment variables for Databricks authentication (using a Personal Access Token) and outlines the necessary `profiles.yml` configuration and dbt CLI commands to initialize a project, test connectivity, and run dbt models on Databricks. Ensure you replace placeholder values with your actual Databricks workspace URL, SQL warehouse ID, and Unity Catalog name (if applicable)."},"warnings":[{"fix":"Always install the `dbt-databricks` version that is explicitly compatible with your `dbt-core` version. Check the official dbt Labs release notes and documentation for compatibility matrices before upgrading.","message":"dbt-databricks adapter versions must align with the `dbt-core` version they are built for. Upgrading `dbt-core` without upgrading `dbt-databricks` (or vice-versa) can lead to unexpected errors, incompatible syntax, or unhandled features.","severity":"breaking","affected_versions":"All versions, particularly across major `dbt-core` upgrades (e.g., from `dbt-core` 1.x to 1.y)."},{"fix":"For Databricks SQL Endpoints, use `/sql/1.0/warehouses/<YOUR_SQL_WAREHOUSE_ID>`. For legacy interactive clusters, use `/sql/protocolv1/o/<YOUR_WORKSPACE_ID>/<YOUR_CLUSTER_ID>`. Ensure the ID is correct and accessible to your user/service principal.","message":"Incorrect `http_path` configuration in `profiles.yml` is a frequent cause of connection failures. The required path varies significantly between Databricks SQL Endpoints (recommended) and older cluster-based connections.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For production environments, prioritize more secure methods like Azure AD Service Principal or OAuth. Ensure environment variables (`DBT_DATABRICKS_TOKEN`, `DBT_DATABRICKS_CLIENT_ID`, `DBT_DATABRICKS_CLIENT_SECRET`) are correctly set and referenced in `profiles.yml` using `{{ env_var('...') }}`.","message":"Databricks authentication methods (Personal Access Tokens via `token`, Azure AD Service Principals via `client_id`/`client_secret`, or OAuth) have different security and configuration requirements. Using PATs is simpler for development but less secure for production. Azure AD Service Principals or OAuth are preferred for production Azure Databricks deployments.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Explicitly set the `catalog` parameter in your `profiles.yml` to the correct Unity Catalog name. Verify that your Databricks user or service principal has the necessary `USE CATALOG`, `USE SCHEMA`, and other required permissions on the target catalog and schema.","message":"When working with Unity Catalog, the `catalog` parameter in `profiles.yml` is crucial. Omitting it or providing an incorrect catalog name can lead to 'schema not found' or 'permission denied' errors, even if the schema exists in a different catalog.","severity":"gotcha","affected_versions":"All versions supporting Unity Catalog"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}