{"id":1328,"library":"apache-airflow-providers-microsoft-fabric","title":"Microsoft Fabric Provider for Apache Airflow","description":"A Python package that helps Data and Analytics engineers trigger on-demand job items of Microsoft Fabric in Apache Airflow DAGs. It enables orchestration of various Fabric items like Notebooks, Pipelines, Spark job definitions, and Semantic Model refreshes. The current version is 0.0.9, and as a newly developed provider, it is expected to have frequent updates.","status":"active","version":"0.0.9","language":"en","source_language":"en","source_url":"https://github.com/microsoft/apache-airflow-microsoft-fabric-plugin.git","tags":["apache-airflow","microsoft","fabric","etl","data-engineering","data-factory","notebooks","pipelines"],"install":[{"cmd":"pip install apache-airflow-providers-microsoft-fabric","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Core dependency for any Apache Airflow provider.","package":"apache-airflow","optional":false},{"reason":"Required for authentication using Microsoft OAuth tokens (implicitly needed for refresh token handling).","package":"msal","optional":false}],"imports":[{"symbol":"MSFabricRunItemOperator","correct":"from airflow.providers.microsoft.fabric.operators.run_item import MSFabricRunItemOperator"}],"quickstart":{"code":"from __future__ import annotations\n\nimport pendulum\n\nfrom airflow.models.dag import DAG\nfrom airflow.providers.microsoft.fabric.operators.run_item import MSFabricRunItemOperator\nimport os\n\nFABRIC_WORKSPACE_ID = os.environ.get('FABRIC_WORKSPACE_ID', 'your_workspace_id')\nFABRIC_ITEM_ID = os.environ.get('FABRIC_ITEM_ID', 'your_item_id') # e.g., Notebook or Pipeline ID\nFABRIC_CONN_ID = os.environ.get('FABRIC_CONN_ID', 'fabric_default') # Airflow Connection ID\n\nwith DAG(\n    dag_id=\"fabric_run_item_example\",\n    start_date=pendulum.datetime(2023, 10, 26, tz=\"UTC\"),\n    catchup=False,\n    schedule=None,\n    tags=[\"microsoft\", \"fabric\", \"etl\"],\n) as dag:\n    run_fabric_notebook = MSFabricRunItemOperator(\n        task_id=\"run_fabric_notebook_task\",\n        workspace_id=FABRIC_WORKSPACE_ID,\n        item_id=FABRIC_ITEM_ID,\n        fabric_conn_id=FABRIC_CONN_ID,\n        job_type=\"RunNotebook\", # or \"Pipeline\", \"SparkJobDefinition\", \"SemanticModel\" etc.\n        wait_for_termination=True,\n        timeout=60 * 60, # 1 hour timeout\n    )\n","lang":"python","description":"A basic Apache Airflow DAG demonstrating how to use the `MSFabricRunItemOperator` to trigger a Microsoft Fabric notebook or pipeline. Ensure your Airflow environment has a 'Generic' connection configured with `fabric_conn_id` containing the Microsoft Entra ID (formerly Azure Active Directory) Service Principal credentials (Client ID in Login, Refresh Token in Password, and Tenant ID/Client Secret/Scopes in Extra)."},"warnings":[{"fix":"Configure your Airflow connection of type 'Generic' precisely following the provider's specific guidelines for Microsoft Entra ID Service Principal authentication. This includes obtaining and managing the OAuth refresh token.","message":"Authentication with Microsoft Fabric APIs requires a 'Generic' connection type in Airflow with specific fields: Client ID in 'Login', Refresh Token in 'Password', and Tenant ID, optional Client Secret, and Scopes in 'Extra'. This deviates from standard Airflow connection forms.","severity":"gotcha","affected_versions":"All versions 0.0.x"},{"fix":"Always refer to the GitHub repository's `README.md` and any available release notes/changelogs before upgrading. Thoroughly test existing DAGs after updates to account for potential breaking changes in operator names or parameters.","message":"The provider is in early development (version 0.0.x), which implies a potential for frequent API changes, renames of modules, operators, and connection parameters without necessarily adhering to strict semantic versioning for minor changes.","severity":"breaking","affected_versions":"All 0.x.x versions"},{"fix":"Exercise caution with `requirements.txt` to prevent dependency conflicts. If environment issues occur, download and manually inspect cluster logs for detailed errors. Consider building dependencies in a local environment first or using simpler dependency sets.","message":"When running on Microsoft Fabric's hosted Apache Airflow, users may encounter difficulties with installing Python packages, debugging environment failures, and recovering from bad installs. The built-in requirements validator might not always detect dependency conflicts, leading to 'unhealthy' scheduler/triggerer states with unhelpful error messages.","severity":"gotcha","affected_versions":"All versions when deployed on Microsoft Fabric's hosted Airflow service."},{"fix":"Establish a process for regularly refreshing the Microsoft OAuth refresh token and updating the corresponding Airflow connection. For Fabric's hosted Airflow, this might involve manual updates via the UI or integration with an external secret management solution.","message":"The Microsoft OAuth refresh tokens used for Service Principal authentication have expiration periods. The generic Airflow connection will need these tokens periodically refreshed to maintain connectivity and prevent DAG failures.","severity":"gotcha","affected_versions":"All versions 0.0.x"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}