{"id":8920,"library":"dagster-airbyte","title":"Dagster Airbyte Integration","description":"The `dagster-airbyte` library provides a robust integration layer to connect Airbyte data synchronization jobs with Dagster's asset-based orchestration. It allows users to define Airbyte connections and syncs as Dagster assets, enabling data lineage tracking, scheduling, and monitoring within the Dagster UI. As part of the Dagster monorepo, its versions are tightly coupled with the core `dagster` library releases. The current version is 0.29.0.","status":"active","version":"0.29.0","language":"en","source_language":"en","source_url":"https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-airbyte","tags":["dagster","airbyte","etl","orchestration","data integration","data pipeline"],"install":[{"cmd":"pip install dagster-airbyte","lang":"bash","label":"Install dagster-airbyte"}],"dependencies":[{"reason":"Core Dagster library required for orchestration and asset definition.","package":"dagster"},{"reason":"Python client for interacting with the Airbyte API.","package":"airbyte-api"}],"imports":[{"symbol":"AirbyteResource","correct":"from dagster_airbyte import AirbyteResource"},{"symbol":"load_assets_from_airbyte_instance","correct":"from dagster_airbyte import load_assets_from_airbyte_instance"},{"note":"Used for more granular control over asset definitions, e.g., custom asset keys.","symbol":"AirbyteSyncAssetFactory","correct":"from dagster_airbyte import AirbyteSyncAssetFactory"}],"quickstart":{"code":"import os\nfrom dagster import Definitions, asset\nfrom dagster_airbyte import AirbyteResource, load_assets_from_airbyte_instance\n\n# Configure Airbyte connection via environment variables for security\nAIRBYTE_HOST = os.environ.get('AIRBYTE_HOST', 'localhost')\nAIRBYTE_PORT = os.environ.get('AIRBYTE_PORT', '8000')\nAIRBYTE_USERNAME = os.environ.get('AIRBYTE_USERNAME', 'airbyte') # Default for local Airbyte\nAIRBYTE_PASSWORD = os.environ.get('AIRBYTE_PASSWORD', 'password') # Default for local Airbyte\n\n# Instantiate the Airbyte resource\nairbyte_resource = AirbyteResource(\n    host=AIRBYTE_HOST,\n    port=AIRBYTE_PORT,\n    username=AIRBYTE_USERNAME,\n    password=AIRBYTE_PASSWORD\n)\n\n# Load assets from your Airbyte instance\n# By default, this will load assets for all connections in Airbyte\nairbyte_assets = load_assets_from_airbyte_instance(airbyte_resource)\n\n# Example of a downstream Dagster asset that depends on an Airbyte asset\n@asset\ndef my_downstream_asset(my_airbyte_table):\n    # 'my_airbyte_table' would be automatically available if an Airbyte connection\n    # produces an asset with this key (e.g., 'airbyte/source_name/table_name')\n    # In a real scenario, you'd specify dependencies more explicitly.\n    print(f\"Processing data from {my_airbyte_table}\")\n    return 'processed_data'\n\n# Combine assets into Dagster Definitions\ndefs = Definitions(\n    assets=[*airbyte_assets, my_downstream_asset],\n    resources={\n        \"airbyte\": airbyte_resource\n    }\n)\n\n# To run this, save as a Python file (e.g., 'airbyte_pipeline.py')\n# Then, from your terminal, navigate to the directory and run:\n# dagster dev -f airbyte_pipeline.py\n","lang":"python","description":"This quickstart demonstrates how to define Airbyte connections as a Dagster resource and load all available Airbyte syncs as Dagster assets using `load_assets_from_airbyte_instance`. It also shows how a Dagster asset can depend on an Airbyte-managed asset. Ensure your Airbyte instance is running and accessible from where Dagster is executed, and configure credentials via environment variables."},"warnings":[{"fix":"Always check the Dagster release notes or `pyproject.toml` for the recommended `dagster-airbyte` version that corresponds to your `dagster` core version. Upgrade both packages concurrently.","message":"Dagster library versions are tightly coupled to core `dagster` versions. Updating `dagster` (e.g., to `1.x.x`) often requires updating `dagster-airbyte` to its corresponding compatible version (e.g., `0.x.x`). Always check the official Dagster release notes for version compatibility.","severity":"breaking","affected_versions":"All versions"},{"fix":"Verify that your Airbyte instance is running and reachable from the Dagster environment. Ensure `AirbyteResource` configuration matches your Airbyte deployment details and authentication settings.","message":"Airbyte integration requires a running and accessible Airbyte instance. The `AirbyteResource`'s `host`, `port`, `username`, and `password` parameters must accurately reflect your Airbyte setup.","severity":"gotcha","affected_versions":"All versions"},{"fix":"For custom asset keys or selective syncs, define assets using `AirbyteSyncAssetFactory` which allows overriding asset key logic and specifying particular connections/streams. Consult the Dagster documentation for `AirbyteSyncAssetFactory` usage.","message":"The `load_assets_from_airbyte_instance` utility automatically generates Dagster asset keys based on Airbyte source and stream names. If you need custom asset keys or more fine-grained control over how Airbyte connections are mapped to Dagster assets, use `AirbyteSyncAssetFactory` instead.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Install the package: `pip install dagster-airbyte`","cause":"The `dagster-airbyte` package is not installed in the Python environment where Dagster is running.","error":"ModuleNotFoundError: No module named 'dagster_airbyte'"},{"fix":"1. Verify the Airbyte instance is actively running. 2. Check the `host` and `port` configured in your `AirbyteResource` to ensure they are correct and reachable from where Dagster is running. 3. Check for firewall rules or network issues.","cause":"Dagster's `AirbyteResource` failed to establish a connection to the specified Airbyte API endpoint. This often means Airbyte is not running or the network path is blocked.","error":"dagster.core.errors.DagsterInvariantViolationError: Could not connect to Airbyte at http://localhost:8000. Please ensure that Airbyte is running and accessible."},{"fix":"Double-check the `AIRBYTE_USERNAME` and `AIRBYTE_PASSWORD` environment variables (or direct resource configuration) against your Airbyte instance's security settings. For local Airbyte, default credentials are often `airbyte`/`password`.","cause":"The credentials (username, password) provided to the `AirbyteResource` are incorrect for authenticating with the Airbyte API.","error":"airbyte_api.errors.UnprocessableEntity: Authentication failed. Please check your username and password."},{"fix":"Ensure all required configuration parameters (`host`, `port`, `username`, `password`) are provided when instantiating `AirbyteResource`. It's recommended to use environment variables for sensitive credentials.","cause":"The `AirbyteResource` was not properly configured with required parameters like `host`, `port`, `username`, or `password`.","error":"dagster._core.errors.DagsterInvalidConfigError: Error validating config for resource 'airbyte': Missing required config field 'host'."}]}