dbt-spark

1.10.1 · active · verified Thu Apr 09

dbt-spark is the Apache Spark adapter plugin for dbt (data build tool), enabling data analysts and engineers to transform data in Apache Spark using SQL. It leverages Spark's distributed computing capabilities for efficient data transformation. The current version is 1.10.1, and it typically releases new versions in alignment with `dbt-core`'s major and minor releases.

Warnings

breaking The minor versions of `dbt-spark` and `dbt-core` must match for correct dependency resolution and functionality (e.g., `dbt-spark==1.9.x` requires `dbt-core==1.9.x`). Mixing versions can lead to errors.
Fix: Always install `dbt-core` and `dbt-spark` with matching minor versions: `pip install dbt-core==X.Y.Z dbt-spark==X.Y.Z`.
deprecated For Databricks users, the `dbt-databricks` adapter is now the recommended choice over `dbt-spark`, offering easier setup, Unity Catalog support, and better defaults. Migration is advised.
Fix: Migrate your project from `dbt-spark` to `dbt-databricks`. Install `dbt-databricks` and update your `profiles.yml` configuration.
gotcha The default `incremental_strategy` for `dbt-spark` is `append`, whereas for the `dbt-databricks` adapter, it defaults to `merge`. This can lead to different behavior in incremental models if migrating or using both adapters.
Fix: Explicitly set `incremental_strategy: 'merge'` or `incremental_strategy: 'append'` in your incremental models to ensure consistent behavior across adapters.
gotcha When connecting to a Spark Thrift server, ensure the target `schema` (database) specified in `profiles.yml` already exists in Spark. If it doesn't, dbt will raise a 'Cannot set database in spark!' runtime error.
Fix: Manually create the database in Spark (e.g., `CREATE DATABASE your_schema_name;`) before running dbt. Also, ensure the `default` namespace exists when using Thrift.
gotcha Using `dbt-spark` with a schema containing a large number of tables (e.g., thousands) can lead to extremely slow `dbt run` parsing times. This is due to Spark's lack of an information schema layer, forcing dbt to 'discover' all tables.
Fix: Consider using dedicated schemas for dbt models with fewer tables or exploring adapter-specific workarounds (if available) for `list_relations_without_caching` limitations.

Install

pip install dbt-core dbt-spark Base Installation
pip install "dbt-spark[ODBC]" For ODBC connections
pip install "dbt-spark[PyHive]" For Thrift or HTTP connections

Imports

dbt-spark

dbt-spark is used primarily via the dbt CLI and configuration files (profiles.yml), not via direct Python import statements in user projects.

dbt adapters are loaded by dbt-core internally; users interact with them through dbt commands and profile configurations.

Quickstart

This quickstart outlines the `profiles.yml` configuration for connecting dbt to a local Spark Thrift server, often set up via docker-compose (as demonstrated in the dbt-spark repository README). It also suggests a basic SQL model for validation.

import os

# This quickstart demonstrates configuring dbt-spark with a local Spark Thrift server.
# First, ensure you have Docker installed and the dbt-spark local environment set up.
# From the dbt-adapters/dbt-spark directory, run:
# docker-compose up -d

# Create a profiles.yml file in your dbt project's ~/.dbt/ directory or project root
profiles_content = '''
spark_local_dev:
  target: dev
  outputs:
    dev:
      type: spark
      method: thrift
      host: 127.0.0.1
      port: 10000
      user: dbt
      schema: analytics
      connect_retries: 5
      connect_timeout: 60
      retry_all: true
'''

# For demonstration, we'll write it to a temporary location
# In a real scenario, this goes to ~/.dbt/profiles.yml
# or in your dbt project folder directly.
profile_path = os.path.expanduser('~/.dbt/profiles.yml') # For a real setup
# Or for a quick test in a temporary project directory:
# profile_path = 'dbt_project/profiles.yml'

# Ensure the directory exists if writing to ~/.dbt/
os.makedirs(os.path.dirname(profile_path), exist_ok=True)

with open(profile_path, 'w') as f:
    f.write(profiles_content)

print(f"profiles.yml created at {profile_path} (or its content suggested for it).")
print("Next, initialize a dbt project: dbt init my_spark_project")
print("Select 'spark_local_dev' as your profile when prompted.")
print("Then, create a model, e.g., models/my_model.sql:")
print("---\nSELECT 1 AS id, 'hello dbt-spark' AS message\n---")
print("Run your dbt models: dbt run --profile spark_local_dev")

view raw JSON →