{"id":1324,"library":"apache-airflow-microsoft-fabric-plugin","title":"Apache Airflow Microsoft Fabric Plugin","description":"The `apache-airflow-microsoft-fabric-plugin` provides an operator and hook to interact with Microsoft Fabric items, such as Lakehouse, Notebook, Datafactory, and Datawarehouse, directly from Apache Airflow. It enables running Spark jobs (like notebooks) within Fabric. The current version is 1.0.3, and it is a community-driven project updated as needed.","status":"active","version":"1.0.3","language":"python","source_language":"en","source_url":"https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin.git","tags":["airflow","microsoft-fabric","etl","data-integration","plugin","azure"],"install":[{"cmd":"pip install apache-airflow-microsoft-fabric-plugin","lang":"bash","label":"Install latest version"}],"dependencies":[{"reason":"Requires Apache Airflow to function, specifically version 2.4.0 or higher.","package":"apache-airflow","optional":false},{"reason":"Used for Microsoft Authentication Library (MSAL) integration to authenticate with Azure/Fabric.","package":"msal","optional":false}],"imports":[{"symbol":"FabricRunSparkJobOperator","correct":"from apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator"},{"symbol":"FabricHook","correct":"from apache_airflow_microsoft_fabric_plugin.hooks.fabric import FabricHook"}],"quickstart":{"code":"from __future__ import annotations\n\nimport os\nimport pendulum\n\nfrom airflow.models.dag import DAG\nfrom apache_airflow_microsoft_fabric_plugin.operators.fabric import FabricRunSparkJobOperator\n\n# Configure your Airflow connection 'azure_fabric_default' with type 'Microsoft Fabric'\n# and extra fields like {\"tenant_id\": \"...\", \"client_id\": \"...\", \"client_secret\": \"...\"}\n# You can use environment variables for sensitive data in real scenarios.\n\nwith DAG(\n    dag_id=\"microsoft_fabric_notebook_execution_dag\",\n    schedule=None,\n    start_date=pendulum.datetime(2023, 10, 26, tz=\"UTC\"),\n    catchup=False,\n    tags=[\"microsoft_fabric\", \"notebook\"],\n) as dag:\n    run_spark_job = FabricRunSparkJobOperator(\n        task_id=\"run_spark_notebook_task\",\n        fabric_conn_id=\"azure_fabric_default\", # Ensure this matches your Airflow connection ID\n        workspace_id=os.environ.get(\"FABRIC_WORKSPACE_ID\", \"your_fabric_workspace_id\"),\n        lakehouse_id=os.environ.get(\"FABRIC_LAKEHOUSE_ID\", \"your_fabric_lakehouse_id\"), # Optional, if notebook interacts with a specific lakehouse\n        notebook_id=os.environ.get(\"FABRIC_NOTEBOOK_ID\", \"your_fabric_notebook_id\"),\n        job_parameters={\n            \"param1\": \"airflow_run\",\n            \"dag_run_id\": \"{{ dag_run.run_id }}\"\n        } # Optional: pass parameters to your notebook\n    )\n","lang":"python","description":"This example DAG demonstrates how to use the `FabricRunSparkJobOperator` to execute a Microsoft Fabric Spark Notebook. Before running, configure an Airflow Connection with `Conn Id: azure_fabric_default` (or your custom ID), `Conn Type: Microsoft Fabric`, and necessary authentication details (e.g., `tenant_id`, `client_id`, `client_secret` for service principal, or `managed_identity_client_id` for Managed Identity) in its `Extra` JSON field. Replace placeholder IDs with your actual Fabric Workspace, Lakehouse (if applicable), and Notebook IDs."},"warnings":[{"fix":"When creating or editing the Airflow Connection for Microsoft Fabric, ensure you select 'Microsoft Fabric' from the 'Conn Type' dropdown in the Airflow UI.","message":"The Airflow Connection must be configured with 'Microsoft Fabric' as the 'Conn Type'. Selecting a generic or incorrect connection type will lead to authentication and API errors.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Verify that your 'Extra' field JSON is valid and contains the correct credentials. For example: `{\"tenant_id\": \"YOUR_TENANT_ID\", \"client_id\": \"YOUR_CLIENT_ID\", \"client_secret\": \"YOUR_CLIENT_SECRET\"}`.","message":"Authentication details (tenant_id, client_id, client_secret for service principal, or managed_identity_client_id for Managed Identity) must be correctly specified in the 'Extra' JSON field of the Airflow Connection.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure your Airflow environment is running version 2.4.0 or newer. Upgrade Airflow if necessary.","message":"This plugin requires Apache Airflow version 2.4.0 or higher. Older Airflow versions may not support the necessary provider interfaces or features.","severity":"gotcha","affected_versions":"<2.4.0"},{"fix":"Install the 'pendulum' package in your environment: `pip install pendulum`.","message":"The 'pendulum' Python package is a required dependency and must be installed in the environment for the library or its Airflow provider to function correctly.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure that Apache Airflow and its relevant provider package (e.g., `apache-airflow-providers-microsoft-fabric`) are correctly installed in your Python environment. For example, `pip install apache-airflow[microsoft.fabric]`.","message":"The required Python package 'pendulum' is not found in the environment. This is a core dependency typically installed with Apache Airflow.","severity":"breaking","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-20T04:27:12.551Z","next_check":"2026-07-08T00:00:00.000Z","problems":[{"fix":"Install the plugin using 'pip install apache-airflow-microsoft-fabric-plugin'.","cause":"The 'apache-airflow-microsoft-fabric-plugin' is not installed or not in the Python path.","error":"ModuleNotFoundError: No module named 'airflow.providers.microsoft.fabric'"},{"fix":"Ensure the correct import path: 'from airflow.providers.microsoft.fabric.hooks.fabric import MicrosoftFabricHook'.","cause":"The import path is incorrect or the module is not installed.","error":"ImportError: cannot import name 'MicrosoftFabricHook' from 'airflow.providers.microsoft.fabric.hooks.fabric'"},{"fix":"Install 'pyspark' using 'pip install pyspark'.","cause":"The 'pyspark' library is not installed in the environment.","error":"ModuleNotFoundError: No module named 'pyspark'"},{"fix":"Install the required Azure package using 'pip install azure'.","cause":"The 'azure' library is not installed in the environment.","error":"ModuleNotFoundError: No module named 'azure'"},{"fix":"Upgrade Airflow to the latest version using 'pip install --upgrade apache-airflow'.","cause":"The 'SUPERVISOR_COMMS' attribute is missing due to an outdated or incompatible Airflow version.","error":"ImportError: cannot import name 'SUPERVISOR_COMMS' from 'airflow.sdk.execution_time.task_runner'"}],"ecosystem":"pypi","meta_description":null,"install_score":0,"install_tag":"stale","quickstart_score":null,"quickstart_tag":null,"pypi_latest":"1.0.3","cli_name":"","cli_version":null,"type":"library","homepage":null,"github":"https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin.git","docs":null,"changelog":null,"pypi":"https://pypi.org/project/apache-airflow-microsoft-fabric-plugin/","npm":null,"openapi_spec":null,"status_page":null,"smithery":null,"categories":["devops","workflow","data","azure"],"install_checks":{"last_tested":"2026-05-20","tag":"stale","tag_description":"widespread failures or data too old to trust","installed_version":"1.0.3","pypi_latest":"1.0.3","is_stale":false,"results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"17.9M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":1.4,"import_time_s":null,"mem_mb":null,"disk_size":"18M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"19.7M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":1.6,"import_time_s":null,"mem_mb":null,"disk_size":"20M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"11.6M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":1.4,"import_time_s":null,"mem_mb":null,"disk_size":"12M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"11.3M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":1.4,"import_time_s":null,"mem_mb":null,"disk_size":"12M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":"17.4M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":0,"wheel_type":"wheel","failure_reason":null,"import_side_effects":"broken","install_time_s":1.7,"import_time_s":null,"mem_mb":null,"disk_size":"18M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"apache-airflow-microsoft-fabric-plugin","exit_code":1,"wheel_type":null,"failure_reason":null,"import_side_effects":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]},"_links":{"self":"https://checklist.day/api/registry/apache-airflow-microsoft-fabric-plugin","v1":"https://checklist.day/v1/registry/apache-airflow-microsoft-fabric-plugin","v1_install":"https://checklist.day/v1/registry/apache-airflow-microsoft-fabric-plugin/install","v1_imports":"https://checklist.day/v1/registry/apache-airflow-microsoft-fabric-plugin/imports","v1_compatibility":"https://checklist.day/v1/registry/apache-airflow-microsoft-fabric-plugin/compatibility","v1_quickstart":"https://checklist.day/v1/registry/apache-airflow-microsoft-fabric-plugin/quickstart","docs":"https://checklist.day/docs"}}