dbt-core

Warnings

• breaking Explicit installation of both `dbt-core` and adapter plugins is now recommended. Prior to v1.8, installing an adapter (e.g., `dbt-snowflake`) implicitly installed `dbt-core`. While this implicit behavior is maintained for backward compatibility currently, it may be removed in future versions.
  Fix: Always explicitly install both `dbt-core` and your desired adapter: `pip install dbt-core dbt-<adapter_name>`.
• gotcha Programmatic `dbtRunner` invocations in the same Python process do not support safe parallel execution. Running multiple dbt commands concurrently within a single process is discouraged due to potential interactions with global Python variables and data platform conflicts.
  Fix: To achieve parallel execution safely, run dbt commands in separate processes (e.g., using `subprocess`, `Celery`) or utilize tools like the dbt CLI or dbt Cloud IDE which manage concurrency.
• deprecated Setting custom global config flags in `profiles.yml` has been deprecated. These flags should now be configured in `dbt_project.yml`.
  Fix: Move global config flags from `profiles.yml` to the `flags` dictionary in `dbt_project.yml`.
• breaking Major versions (e.g., v1 to v2) of dbt Core may introduce breaking changes, including the removal of deprecated functionality. Minor versions (e.g., v1.8 to v1.9) are generally backward compatible for documented features.
  Fix: Always consult the official dbt Developer Hub migration guides and changelogs before upgrading to a new major version.
• deprecated Beginning in v1.10, dbt Core will issue deprecation warnings for invalid dbt code, including custom inputs, duplicate YAML keys, and unexpected Jinja blocks, which will become invalid in future versions.
  Fix: Review and address deprecation warnings promptly. Ensure `dbt_project.yml` and YAML property files adhere to recommended structures and avoid standalone anchor definitions at the top level.
• gotcha dbt Python models (`.py` files) are executed remotely on the configured data platform (e.g., Snowflake, BigQuery, Databricks), not on the local machine where `dbt-core` is running. This means any Python code must be compatible with the platform's execution environment and its limitations (e.g., external API calls might not be supported).
  Fix: Be aware of your data platform's Python runtime capabilities and limitations when developing dbt Python models. Do not assume local Python environment features are available remotely.

Install

• pip install dbt-core Install dbt-core
• pip install dbt-core dbt-<adapter_name> Install dbt-core with a specific adapter (e.g., dbt-snowflake)

Imports

• dbtRunner

  from dbt.cli.main import dbtRunner

  This is the primary programmatic entry point for invoking dbt CLI commands within Python. Available since dbt Core v1.5.
• dbtRunnerResult

  from dbt.cli.main import dbtRunnerResult

  Used to inspect the results of a dbtRunner invocation.

Quickstart

This quickstart demonstrates how to programmatically invoke dbt CLI commands using the `dbtRunner` class. It shows how to run all models in an existing dbt project and process the results. A `dbt_project.yml` and `profiles.yml` must be set up for this to execute successfully. The `dbtRunner` class was introduced in dbt Core v1.5 as the official Python entry point.

import os
from dbt.cli.main import dbtRunner, dbtRunnerResult

# NOTE: For this to run, you must have a dbt project initialized
# (e.g., with 'dbt init <project_name>') and a profiles.yml configured.
# Set the DBT_PROFILES_DIR environment variable or ensure 'dbt_project.yml'
# is in the current working directory or a parent directory.
# A simple dbt_project.yml might look like:
# name: my_dbt_project
# version: '1.0.0'
# config-version: 2
# profile: my_profile

# Initialize the dbt runner
dbt = dbtRunner()

# Example: Run all models in your dbt project
# You might need to set the project directory explicitly if not in CWD
project_directory = os.environ.get('DBT_PROJECT_DIR', os.getcwd())
cli_args = ["run", "--project-dir", project_directory]

print(f"Invoking dbt run with arguments: {cli_args}")
res: dbtRunnerResult = dbt.invoke(cli_args)

if res.success:
    print("dbt run completed successfully.")
    if res.result:
        for r in res.result:
            print(f"  Model: {r.node.name}, Status: {r.status}")
else:
    print("dbt run failed.")
    if res.exception:
        print(f"Exception: {res.exception}")
    elif res.result:
        # Some commands might return partial results even on failure
        print("Partial results or errors:")
        for r in res.result:
            print(f"  Model: {r.node.name}, Status: {r.status}, Message: {r.message}")