{"id":1324,"library":"apache-airflow-microsoft-fabric-plugin","title":"Apache Airflow Microsoft Fabric Plugin","description":"The `apache-airflow-microsoft-fabric-plugin` provides an operator and hook to interact with Microsoft Fabric items, such as Lakehouse, Notebook, Datafactory, and Datawarehouse, directly from Apache Airflow. It enables running Spark jobs (like notebooks) within Fabric. The current version is 1.0.3, and it is a community-driven project updated as needed.","status":"active","version":"1.0.3","language":"en","source_language":"en","source_url":"https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin.git","tags":["airflow","microsoft-fabric","etl","data-integration","plugin","azure"],"install":[{"cmd":"pip install apache-airflow-microsoft-fabric-plugin","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Requires Apache Airflow to function, specifically version 2.4.0 or higher.","package":"apache-airflow","optional":false},{"reason":"Used for Microsoft Authentication Library (MSAL) integration to authenticate with Azure/Fabric.","package":"msal","optional":false}],"imports":[{"symbol":"FabricRunSparkJobOperator","correct":"from apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator"},{"symbol":"FabricHook","correct":"from apache_airflow_microsoft_fabric_plugin.hooks.fabric import FabricHook"}],"quickstart":{"code":"from __future__ import annotations\n\nimport os\nimport pendulum\n\nfrom airflow.models.dag import DAG\nfrom apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator\n\n# Configure your Airflow connection 'azure_fabric_default' with type 'Microsoft Fabric'\n# and extra fields like {\"tenant_id\": \"...\", \"client_id\": \"...\", \"client_secret\": \"...\"}\n# You can use environment variables for sensitive data in real scenarios.\n\nwith DAG(\n    dag_id=\"microsoft_fabric_notebook_execution_dag\",\n    schedule=None,\n    start_date=pendulum.datetime(2023, 10, 26, tz=\"UTC\"),\n    catchup=False,\n    tags=[\"microsoft_fabric\", \"notebook\"],\n) as dag:\n    run_spark_job = FabricRunSparkJobOperator(\n        task_id=\"run_spark_notebook_task\",\n        fabric_conn_id=\"azure_fabric_default\", # Ensure this matches your Airflow connection ID\n        workspace_id=os.environ.get(\"FABRIC_WORKSPACE_ID\", \"your_fabric_workspace_id\"),\n        lakehouse_id=os.environ.get(\"FABRIC_LAKEHOUSE_ID\", \"your_fabric_lakehouse_id\"), # Optional, if notebook interacts with a specific lakehouse\n        notebook_id=os.environ.get(\"FABRIC_NOTEBOOK_ID\", \"your_fabric_notebook_id\"),\n        job_parameters={\n            \"param1\": \"airflow_run\",\n            \"dag_run_id\": \"{{ dag_run.run_id }}\"\n        } # Optional: pass parameters to your notebook\n    )\n","lang":"python","description":"This example DAG demonstrates how to use the `FabricRunSparkJobOperator` to execute a Microsoft Fabric Spark Notebook. Before running, configure an Airflow Connection with `Conn Id: azure_fabric_default` (or your custom ID), `Conn Type: Microsoft Fabric`, and necessary authentication details (e.g., `tenant_id`, `client_id`, `client_secret` for service principal, or `managed_identity_client_id` for Managed Identity) in its `Extra` JSON field. Replace placeholder IDs with your actual Fabric Workspace, Lakehouse (if applicable), and Notebook IDs."},"warnings":[{"fix":"When creating or editing the Airflow Connection for Microsoft Fabric, ensure you select 'Microsoft Fabric' from the 'Conn Type' dropdown in the Airflow UI.","message":"The Airflow Connection must be configured with 'Microsoft Fabric' as the 'Conn Type'. Selecting a generic or incorrect connection type will lead to authentication and API errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Verify that your 'Extra' field JSON is valid and contains the correct credentials. For example: `{\"tenant_id\": \"YOUR_TENANT_ID\", \"client_id\": \"YOUR_CLIENT_ID\", \"client_secret\": \"YOUR_CLIENT_SECRET\"}`.","message":"Authentication details (tenant_id, client_id, client_secret for service principal, or managed_identity_client_id for Managed Identity) must be correctly specified in the 'Extra' JSON field of the Airflow Connection.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your Airflow environment is running version 2.4.0 or newer. Upgrade Airflow if necessary.","message":"This plugin requires Apache Airflow version 2.4.0 or higher. Older Airflow versions may not support the necessary provider interfaces or features.","severity":"gotcha","affected_versions":"<2.4.0"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}