dbt-loom

0.9.4 · active · verified Thu Apr 16

dbt-loom is a dbt-core plugin designed to facilitate multi-project deployments by enabling the injection of public model definitions from upstream dbt artifacts into downstream dbt projects. It supports various sources for these artifacts, including local files, remote HTTP(S) endpoints, dbt Cloud, and major object storage providers like S3, Google Cloud Storage, and Azure Storage. The library is currently at version 0.9.4 and is under active development, receiving regular updates to enhance features and maintain compatibility with dbt-core.

Common errors

```
Compilation Error 'model.parent.dim_commission_type' depends on 'seed.parent.seed_commission_type' which is not in the graph!
```
cause This typically occurs when a public model in an upstream project depends on a protected (non-publicly exposed) seed in the same upstream project, and the downstream project cannot resolve the seed dependency.

fix Ensure that any seeds or other internal resources that are dependencies of public models in an upstream project are also made accessible or handled appropriately within the upstream project's manifest, or restructure dependencies to avoid this scenario.
```
Compilation Error 'model.revenue.orders.v1' which is not in the graph! (or similar with versioned models like 'model.revenue.orders.v1.0')
```
cause This error can occur when working with versioned models, potentially due to inconsistencies in how `dbt-core` versions represent node unique IDs for versioned models, or lingering artifacts from previous `dbt` runs.

fix Try running `dbt clean` in both the upstream and downstream projects before `dbt deps` and `dbt build`/`dbt run`. This can clear out incompatible unique_ids generated between `dbt-core` versions. Also, verify `dbt-loom` and `dbt-core` versions for known compatibilities.
```
Parsing Error Node model.great_bay.test_us_population attempted to reference node model.balboa.us_population, which is not allowed because the referenced node is private to the marketing group.
```
cause Attempting to reference a dbt model in an upstream project that has not been explicitly marked with `access: public` in its `schema.yml` file. dbt-loom purposefully only injects public models.

fix Update the upstream project's `schema.yml` file to set `access: public` for the `model.balboa.us_population` model if it is intended to be consumed by other projects.

Warnings

gotcha dbt-core's plugin API, which dbt-loom utilizes, is still in beta. This means that future updates to dbt-core may introduce breaking changes to the plugin interface, potentially requiring updates to dbt-loom.
Fix: Monitor dbt-loom and dbt-core release notes for compatibility updates. Pin dbt-loom and dbt-core versions in your `requirements.txt`.
gotcha Documentation generated by `dbt docs generate` for models injected by dbt-loom may be sparse. This is because `PluginNodeArgs` (how dbt-loom injects nodes) are not fully-realized `dbt ManifestNode` objects.
Fix: Be aware of this limitation when relying on dbt-generated documentation for cross-project models. Refer to the upstream project's documentation directly for full details.
breaking dbt-loom requires `dbt-core` version 1.6.0-b8 or newer. For specific features, such as fetching manifest files from Snowflake Stage or Databricks Volumes/DBFS/Workspace, `dbt-core` version 1.8.0 or newer is required.
Fix: Upgrade your `dbt-core` installation and compatible dbt adapter to at least 1.6.0-b8, and 1.8.0+ if using advanced warehouse storage manifest sources.
gotcha Only dbt models explicitly marked with `access: public` in their `schema.yml` file in the upstream project will be injected into downstream projects by dbt-loom.
Fix: Ensure all models intended for cross-project consumption are explicitly marked as `access: public` in the upstream dbt project's schema definition.

Install

pip install dbt-loom Install dbt-loom

Imports

dbt-loom

dbt-loom functionality is typically configured via dbt_loom.config.yml and hooks into dbt-core's plugin system. Direct Python imports are not usually required by end-users.

dbt-loom operates as a dbt-core plugin, activated through its configuration file (dbt_loom.config.yml) rather than explicit Python imports in user code.

Quickstart

To quickly get started with dbt-loom, first install the package. Then, ensure your upstream dbt project has `access: public` defined for models you wish to share and generate its `manifest.json`. In your downstream dbt project, create a `dbt_loom.config.yml` file pointing to this upstream manifest. Finally, you can reference the public models using the standard `ref('upstream_project_name', 'model_name')` syntax and run your downstream dbt commands.

# 1. Install dbt-loom
pip install dbt-loom

# 2. Ensure your upstream dbt project has public models defined in its schema.yml
#    e.g., in upstream_project/models/schema.yml:
#    models:
#      - name: public_customers
#        access: public

# 3. Run your upstream dbt project to generate its manifest.json
#    cd upstream_project && dbt build --target production

# 4. Create a dbt_loom.config.yml file in your downstream dbt project's root directory
#    (replace with actual path to upstream manifest.json)
#    Example dbt_loom.config.yml:
#    manifests:
#      - name: upstream_project
#        type: file
#        config:
#          path: ../path/to/upstream_project/target/manifest.json

# 5. Reference the upstream public model in your downstream dbt project
#    e.g., in downstream_project/models/my_downstream_model.sql:
#    SELECT *
#    FROM {{ ref('upstream_project', 'public_customers') }}

# 6. Run your downstream dbt project
#    cd downstream_project && dbt build

view raw JSON →