dagster-fivetran

0.29.3 verified Fri May 01 auth: no python

Package for integrating Fivetran with Dagster. Version 0.29.3; part of the Dagster ecosystem (1.13.3 core). Follows Dagster's release cadence (~monthly).

pip install dagster-fivetran

Common errors

error ImportError: cannot import name 'FivetranResource' from 'dagster_fivetran' ↓

cause Using an older version of dagster-fivetran (<0.23.0) that uses `fivetran_resource` instead.

fix

Upgrade dagster-fivetran: pip install dagster-fivetran --upgrade and use from dagster_fivetran import FivetranResource.

error dagster._core.errors.DagsterInvalidInvocationError: Resource 'fivetran' is not provided. ↓

cause The Fivetran resource is not included in the Definitions `resources` dict.

fix

Add resources={"fivetran": fivetran_instance} to your Definitions.

error dagster_fivetran.errors.FivetranApiError: 401 Client Error: Unauthorized ↓

cause Invalid Fivetran API credentials (account ID, API key, or secret).

fix

Check that FIVETRAN_ACCOUNT_ID, FIVETRAN_API_KEY, FIVETRAN_API_SECRET environment variables are set correctly.

Warnings

breaking In dagster-fivetran 0.23.0+, the `fivetran_resource` function was replaced with the `FivetranResource` class. `fivetran_resource` is deprecated and will be removed. ↓

fix Use `FivetranResource` class with `account_id`, `api_key`, `api_secret` parameters instead of `fivetran_resource`.

breaking In dagster-fivetran 0.20.0+, the `FivetranOutput` and related types changed. Use `FivetranConnectorMetadata` instead. ↓

fix Update type annotations to use the new metadata types.

gotcha Fivetran connectors are loaded as assets by default, but only connectors with `sync_frequency = 0` (manually triggered syncs) are materializable. Connectors with scheduled syncs will not be materializable via Dagster. ↓

fix Set `sync_frequency` to `null` or `0` in Fivetran UI for connectors you want to manage via Dagster, or use `load_assets_from_fivetran_instance` with the `connector_filter` parameter to filter.

deprecated `load_assets_from_fivetran_instance` is being phased out in favor of `build_fivetran_assets_definitions` which provides more flexibility. ↓

fix Use `build_fivetran_assets_definitions` instead of `load_assets_from_fivetran_instance` for new code.

gotcha Fivetran API key and secret are sensitive. Do not hardcode them; use `EnvVar` or environment variables. ↓

fix Use `EnvVar("FIVETRAN_API_KEY")` or `os.environ["FIVETRAN_API_KEY"]` instead of hardcoded values.

Imports

FivetranResource
wrong
```
from dagster_fivetran.resources import FivetranResource
```
correct
```
from dagster_fivetran import FivetranResource
```
FivetranResource is a top-level export; importing from submodule may break in future versions.
fivetran_resource
```
from dagster_fivetran import fivetran_resource
```
Deprecated alias for FivetranResource; use FivetranResource going forward.
load_assets_from_fivetran_instance
```
from dagster_fivetran import load_assets_from_fivetran_instance
```
Commonly used for asset discovery.

build_fivetran_assets_definitions

wrong

from dagster_fivetran.assets import build_fivetran_assets_definitions

correct

from dagster_fivetran import build_fivetran_assets_definitions

build_fivetran_assets_definitions is a top-level export.

Quickstart

Define a Fivetran resource and load all connectors as Dagster assets.

import os
from dagster import AssetExecutionContext, Definitions, EnvVar
from dagster_fivetran import FivetranResource, load_assets_from_fivetran_instance

fivetran_instance = FivetranResource(
    account_id=EnvVar("FIVETRAN_ACCOUNT_ID"),
    api_key=EnvVar("FIVETRAN_API_KEY"),
    api_secret=EnvVar("FIVETRAN_API_SECRET"),
)

# Load all Fivetran connectors as assets
fivetran_assets = load_assets_from_fivetran_instance(
    fivetran_instance,
    connector_to_group_fn=lambda conn: "fivetran",
)

defs = Definitions(
    assets=[fivetran_assets],
    resources={"fivetran": fivetran_instance},
)