dagster-embedded-elt

0.29.1 verified Fri May 01 auth: no python

A Dagster library for performing ETL/ELT tasks, providing integrations with tools like dbt, Sling, and Airbyte. This version (0.29.1) is part of the Dagster 1.13.1 release, compatible with Python 3.10-3.14. The library follows Dagster's release cadence with monthly minor releases and patch releases as needed.

pip install dagster-embedded-elt

Common errors

error ModuleNotFoundError: No module named 'dagster_embedded_elt' ↓

cause The package `dagster-embedded-elt` is not installed.

fix

Run pip install dagster-embedded-elt.

error ImportError: cannot import name 'build_sling_asset' from 'dagster_embedded_elt' ↓

cause Incorrect import path; `build_sling_asset` is in `dagster_embedded_elt.sling`.

fix

Use from dagster_embedded_elt.sling import build_sling_asset.

error dagster._core.errors.DagsterInvalidConfigError: Missing required config entry at root ↓

cause The `SlingResource` requires a `connection_string` config. When using `build_sling_asset`, the source/target connections must be provided either via env vars or config.

fix

Set environment variables (e.g., SOURCE_CONN, TARGET_CONN) or pass connection_string in the replication config under source/target.

Warnings

breaking In dagster-embedded-elt versions prior to 0.29.0, the Sling resource used `SlingResource` from `dagster_embedded_elt.sling.resources`. In 0.29.0, the import path changed to `dagster_embedded_elt.sling`. Update imports to avoid breaking changes. ↓

fix Replace `from dagster_embedded_elt.sling.resources import SlingResource` with `from dagster_embedded_elt.sling import SlingResource`.

deprecated The `build_sling_asset` function's `mode` parameter in the replication config is deprecated in favor of `replication_mode`. The old `mode` key will be removed in a future version. ↓

fix Use `replication_mode` instead of `mode` in your replication config streams.

gotcha When using `build_sling_asset` with a `SlingResource`, ensure you pass the resource in the Definitions. The asset will fail at runtime if the resource key does not match. ↓

fix Always include `resources={"sling": sling_resource}` in your Definitions, and use the default resource key 'sling' or set `resource_key` in `build_sling_asset`.

breaking The `EmbeddedEltDbtResource` has been renamed from `DbtCliResource` in dagster-embedded-elt <=0.27.0. Existing code must be updated. ↓

fix Replace `from dagster_embedded_elt.dbt import DbtCliResource` with `from dagster_embedded_elt.dbt import EmbeddedEltDbtResource`.

Install

pip install dagster-embedded-elt[sling]

pip install dagster-embedded-elt[dbt]

Imports

EmbeddedEltDbtResource
```
from dagster_embedded_elt.dbt import EmbeddedEltDbtResource
```
dbt resource for running dbt projects within Dagster

SlingResource

from dagster_embedded_elt.sling import SlingResource

Sling resource for ELT tasks

build_sling_asset
```
from dagster_embedded_elt.sling import build_sling_asset
```
Helper to create assets from Sling replication configs

Quickstart

Define a Sling-based ELT asset that copies data from Postgres to Snowflake.

from dagster import AssetExecutionContext, Definitions
from dagster_embedded_elt.sling import build_sling_asset, SlingResource

replication_config = {
    "source": {
        "type": "postgres",
        "connection_string": "postgresql://user:pass@localhost:5432/source"
    },
    "target": {
        "type": "snowflake",
        "connection_string": "snowflake://user:pass@account/db/schema?warehouse=wh&role=role"
    },
    "streams": {
        "public.my_table": {
            "object": "my_schema.my_table",
            "mode": "full_refresh"
        }
    }
}

sling_resource = SlingResource()
my_asset = build_sling_asset(
    asset_key="my_table",
    source_connection_string=os.environ.get("SOURCE_CONN", ""),
    target_connection_string=os.environ.get("TARGET_CONN", ""),
    replication_config=replication_config
)

defs = Definitions(assets=[my_asset], resources={"sling": sling_resource})