OpenLineage dbt Integration

1.46.0 · active · verified Fri Apr 17

The `openlineage-dbt` library provides a seamless integration between dbt (data build tool) and OpenLineage, capturing detailed data lineage for dbt models, sources, and exposures. It functions as a dbt adapter, translating dbt operations into OpenLineage events. The current version is 1.46.0 and it typically releases updates to align with new OpenLineage Python client versions and dbt-core major/minor versions, maintaining an active development cadence.

Common errors

```
ModuleNotFoundError: No module named 'openlineage_dbt'
```
cause `openlineage-dbt` is not installed in the current Python environment.

fix Run `pip install openlineage-dbt` to install the package.
```
dbt encountered an error: The 'adapter' field in your profile is missing or invalid. Received adapter type: openlineage
```
cause Your `profiles.yml` configuration for the `openlineage` type is missing the nested `adapter` field, which specifies the actual dbt adapter (e.g., snowflake, bigquery) to wrap.

fix In your `profiles.yml`, under your `type: openlineage` profile, add `adapter: <your_original_dbt_adapter_type>` (e.g., `adapter: snowflake`) along with the necessary connection details for that adapter.
```
Error: Could not connect to OpenLineage API at [URL] - Status: [HTTP Status Code]
```
cause The `OPENLINEAGE_URL` is incorrect, the OpenLineage backend is unreachable, or the `OPENLINEAGE_API_KEY` (if used) is invalid.

fix Verify the `OPENLINEAGE_URL` environment variable or CLI flag is correct and reachable. Check your OpenLineage backend status. If using an API key, ensure `OPENLINEAGE_API_KEY` is correctly set.

Warnings

gotcha Dbt version compatibility is crucial. `openlineage-dbt` is designed to work with specific versions of `dbt-core`. Ensure your `dbt-core` and underlying `dbt-<adapter>` versions align with the `openlineage-dbt` requirements (e.g., `dbt-core>=1.0.0,<2.0.0` for recent `openlineage-dbt` versions).
Fix: Check the `openlineage-dbt` PyPI page or GitHub README for supported `dbt-core` versions and upgrade/downgrade dbt components accordingly.
breaking The `profiles.yml` configuration requires `type: openlineage` and a nested `adapter: <your_dbt_adapter>` field. Omitting or incorrectly configuring the `adapter` field under `type: openlineage` will cause dbt to fail with profile errors.
Fix: Ensure your `profiles.yml` has the `type: openlineage` and `adapter: <your_original_dbt_adapter_type>` fields correctly specified, along with the full connection details for the underlying adapter.
gotcha OpenLineage URL and API Key are mandatory for sending lineage events. If `OPENLINEAGE_URL` is not provided (via environment variable or `--openlineage-url` CLI flag), `openlineage-dbt` will fail to send events or cause dbt runs to error out.
Fix: Set `OPENLINEAGE_URL` and `OPENLINEAGE_API_KEY` (if required by your backend) as environment variables or pass them as CLI flags (`--openlineage-url`, `--openlineage-api-key`) during your dbt command execution.

Install

pip install openlineage-dbt Install core package
pip install 'openlineage-dbt[snowflake]' # or [bigquery], [postgres], etc. Install with dbt adapter (example: Snowflake)

Imports

__version__
```
from openlineage_dbt import __version__
```
This library primarily integrates via dbt CLI commands and `profiles.yml` configuration, rather than direct Python imports for end-user applications. This import is provided as a technical placeholder from the package root.

Quickstart

To quickly integrate `openlineage-dbt`, you need to install the package along with your specific dbt adapter (e.g., `dbt-snowflake`). Then, configure your `profiles.yml` to use `type: openlineage` and specify the `adapter` it should wrap. Finally, set the `OPENLINEAGE_URL` and `OPENLINEAGE_API_KEY` (if required) environment variables or pass them as CLI arguments when running dbt commands. Lineage events will be sent to the configured OpenLineage backend.

# 1. Install openlineage-dbt and your target dbt adapter (e.g., dbt-snowflake)
# pip install openlineage-dbt dbt-snowflake

# 2. Configure your dbt profiles.yml
# (e.g., ~/.dbt/profiles.yml)
# Add an OpenLineage profile that wraps your actual adapter:
# my_openlineage_profile:
#   target: dev
#   outputs:
#     dev:
#       type: openlineage
#       adapter: snowflake # The actual dbt adapter you are using
#       account: <your_snowflake_account>
#       user: <your_snowflake_user>
#       password: <your_snowflake_password>
#       role: <your_snowflake_role>
#       warehouse: <your_snowflake_warehouse>
#       database: <your_snowflake_database>
#       schema: <your_snowflake_schema>
#       threads: 1

# 3. Set OpenLineage URL (and optionally API Key) as environment variables
#    or pass as CLI arguments.
import os
os.environ['OPENLINEAGE_URL'] = os.environ.get('OPENLINEAGE_URL', 'http://localhost:5000') # Or your Marque URL
os.environ['OPENLINEAGE_API_KEY'] = os.environ.get('OPENLINEAGE_API_KEY', 'your_api_key_if_needed')

# 4. Run dbt with the OpenLineage-enabled profile
# Create a dummy dbt project structure for demonstration
# (assuming you have a dbt project initialized, e.g., 'dbt init my_project')
# For example, create models/example.sql in your dbt project:
# -- models/example.sql
# -- {{ config(materialized='table') }}
# -- select 1 as id

print(f"Running dbt with OpenLineage at {os.environ['OPENLINEAGE_URL']}...")

# Example dbt command (execute from your dbt project root):
# dbt build --profile my_openlineage_profile --target dev
# Or, to ensure lineage is sent:
# dbt build --profile my_openlineage_profile --target dev \
#           --openlineage-url $OPENLINEAGE_URL \
#           --openlineage-api-key $OPENLINEAGE_API_KEY

print("Please execute the dbt command from your terminal to see lineage generation.")
print("Example: dbt build --profile my_openlineage_profile --target dev")

view raw JSON →