{"id":6974,"library":"airflow-mcd","title":"Monte Carlo Data Airflow Provider","description":"Monte Carlo's Apache Airflow Provider integrates Monte Carlo with Airflow to enable data observability features like incident alerts, lineage visibility, and pipeline control via Circuit Breakers. It is compatible with Apache Airflow 1.10.14 or greater and requires Python 3.7 or greater. The library is actively maintained with regular updates.","status":"active","version":"0.3.11","language":"en","source_language":"en","source_url":"https://docs.getmontecarlo.com/docs/airflow-incidents-dags-and-tasks","tags":["airflow","apache-airflow","data-observability","data-quality","etl","monitoring","callbacks","dbt","circuit-breaker"],"install":[{"cmd":"pip install airflow-mcd","lang":"bash","label":"Install `airflow-mcd`"},{"cmd":"pip install \"apache-airflow>=1.10.14\" airflow-mcd","lang":"bash","label":"Install `airflow-mcd` alongside Apache Airflow"}],"dependencies":[{"reason":"This package is an Apache Airflow provider, requiring Airflow for functionality.","package":"apache-airflow","optional":false}],"imports":[{"note":"Used for adding broad or explicit Monte Carlo callbacks to DAGs and tasks.","symbol":"mcd_callbacks","correct":"from airflow_mcd.callbacks import mcd_callbacks"},{"note":"An example of a dbt operator provided by Monte Carlo for dbt command execution and artifact sending.","symbol":"DbtRunOperator","correct":"from airflow_mcd.operators.dbt import DbtRunOperator"},{"note":"Used to execute circuit breaker rules based on Monte Carlo monitors, failing downstream tasks if conditions are breached.","symbol":"SimpleCircuitBreakerOperator","correct":"from airflow_mcd.operators.mcd import SimpleCircuitBreakerOperator"}],"quickstart":{"code":"from __future__ import annotations\n\nimport pendulum\n\nfrom airflow.models.dag import DAG\nfrom airflow.operators.bash import BashOperator\n\nfrom airflow_mcd.callbacks import mcd_callbacks\n\nwith DAG(\n    dag_id=\"monte_carlo_example_dag\",\n    start_date=pendulum.datetime(2023, 1, 1, tz=\"UTC\"),\n    catchup=False,\n    schedule=None,\n    tags=[\"monte_carlo\", \"example\"],\n    **mcd_callbacks.dag_callbacks, # Apply broad DAG-level callbacks\n) as dag:\n    start_task = BashOperator(\n        task_id=\"start_task\",\n        bash_command=\"echo 'Starting DAG'\",\n        **mcd_callbacks.task_callbacks, # Apply broad Task-level callbacks\n    )\n\n    process_data = BashOperator(\n        task_id=\"process_data\",\n        bash_command=\"echo 'Processing data...'; sleep 5\",\n        # You can override specific callbacks if needed:\n        # on_failure_callback=mcd_callbacks.mcd_task_failure_callback,\n        **mcd_callbacks.task_callbacks,\n    )\n\n    end_task = BashOperator(\n        task_id=\"end_task\",\n        bash_command=\"echo 'DAG finished'\",\n        **mcd_callbacks.task_callbacks,\n    )\n\n    start_task >> process_data >> end_task\n","lang":"python","description":"This quickstart demonstrates how to integrate Monte Carlo callbacks into an Apache Airflow DAG. The `mcd_callbacks.dag_callbacks` and `mcd_callbacks.task_callbacks` are used to automatically send webhooks to Monte Carlo upon various DAG and task events (success, failure, etc.), providing observability. For full functionality, Monte Carlo API credentials must be configured as an Airflow connection."},"warnings":[{"fix":"Upgrade `airflow-mcd` to 0.3.3 or later: `pip install -U airflow-mcd`.","message":"When upgrading to Airflow 2.9.0 or later, ensure `airflow-mcd` is updated to version 0.3.3 or higher. Older versions of `airflow-mcd` are not compatible with Airflow 2.9.0+ and may cause connection failures.","severity":"breaking","affected_versions":"<0.3.3 (with Airflow >= 2.9.0)"},{"fix":"Always include `**mcd_callbacks.dag_callbacks` (or explicit DAG callbacks) in your DAG definition if you are using any task-level callbacks.","message":"If task-level callbacks are configured, DAG-level callbacks (e.g., `mcd_callbacks.dag_callbacks`) must also be defined. Without DAG callbacks, Monte Carlo cannot update the DAG run status, leading to DAGs appearing perpetually 'Running' and preventing failure alerts.","severity":"gotcha","affected_versions":"All"},{"fix":"Create an Airflow connection of type `Monte Carlo Data` for Circuit Breakers, ensuring the API Key ID is in the 'Login' field and the API Secret in the 'Password' field.","message":"The `Monte Carlo Data Gateway` connection type in Airflow does not support Circuit Breakers operations. For Circuit Breaker functionality, you must use the `Monte Carlo Data` connection type configured with an API Key ID (login) and API Secret (password).","severity":"gotcha","affected_versions":"All"},{"fix":"Upgrade `airflow-mcd` to 0.3.10 or later: `pip install -U airflow-mcd`.","message":"For Airflow 3.0 compatibility, `airflow-mcd` package version 0.3.10 and later is required. Earlier versions of `airflow-mcd` may not function correctly with Airflow 3.0 onwards.","severity":"breaking","affected_versions":"<0.3.10 (with Airflow >= 3.0)"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure that `mcd_callbacks.dag_callbacks` (or explicit DAG-level success/failure callbacks) are included in your DAG definition, in addition to any task-level callbacks.","cause":"This typically indicates that DAG-level callbacks are not properly configured, or are missing, even if task-level callbacks are present.","error":"DAG runs remain in a “running” state indefinitely or no alerts are triggered when a DAG fails in Monte Carlo."},{"fix":"As a workaround, use the generic HTTP connection type. Set `https://integrations.getmontecarlo.com` as the 'Host' and configure your Monte Carlo API ID as 'Login' and API token as 'Password'. Note that HTTP connections cannot be tested within the Airflow UI.","cause":"In some MWAA cases, PyPI packages are not installed in the web server, making custom connection types unavailable.","error":"Airflow connection tests fail or the 'Monte Carlo Data Gateway' connection type is not available in Apache Airflow Managed Workflows for Apache Airflow (MWAA)."},{"fix":"Upgrade the `airflow-mcd` package to version 0.3.3 or later: `pip install -U airflow-mcd`.","cause":"Older versions of `airflow-mcd` (prior to 0.3.3) have known compatibility issues with Airflow 2.9.0+.","error":"Connection to Monte Carlo fails after upgrading Apache Airflow to 2.9.0 or higher."}]}