{"id":968,"library":"apache-airflow","title":"Apache Airflow","description":"Apache Airflow is an open-source platform used to programmatically author, schedule, and monitor workflows, particularly for data pipelines. It defines workflows as Directed Acyclic Graphs (DAGs) in Python, enabling dynamic, scalable, and extensible orchestration. The current stable version is 3.1.8, with releases occurring regularly to introduce new features, improvements, and bug fixes.","status":"active","version":"3.1.8","language":"python","source_language":"en","source_url":"https://github.com/apache/airflow","tags":["workflow orchestration","ETL","data pipelines","scheduler","DAG","Apache Software Foundation"],"install":[{"cmd":"pip install \"apache-airflow[celery,cncf.kubernetes,http,postgres,amazon]\"==3.1.8 --constraint \"https://raw.githubusercontent.com/apache/airflow/constraints-3.1.8/constraints-3.10.txt\"","lang":"bash","label":"Full installation with common extras (PostgreSQL, Celery, Kubernetes, AWS, HTTP) for Python 3.10"},{"cmd":"pip install apache-airflow==3.1.8 --constraint \"https://raw.githubusercontent.com/apache/airflow/constraints-3.1.8/constraints-3.10.txt\"","lang":"bash","label":"Minimal installation for Python 3.10"}],"dependencies":[{"reason":"Required for KubernetesExecutor or KubernetesPodOperator.","package":"apache-airflow-providers-cncf-kubernetes","optional":true},{"reason":"Required for CeleryExecutor.","package":"apache-airflow-providers-celery","optional":true},{"reason":"Provides operators and hooks for AWS services.","package":"apache-airflow-providers-amazon","optional":true},{"reason":"Required for PostgreSQL backend database.","package":"apache-airflow-providers-postgres","optional":true},{"reason":"Provides HttpOperator and HttpHook for interacting with HTTP APIs.","package":"apache-airflow-providers-http","optional":true}],"imports":[{"symbol":"DAG","correct":"from airflow.models.dag import DAG"},{"symbol":"BashOperator","correct":"from airflow.operators.bash import BashOperator"},{"symbol":"PythonOperator","correct":"from airflow.operators.python import PythonOperator"},{"symbol":"TaskGroup","correct":"from airflow.utils.task_group import TaskGroup"},{"note":"Operators specific to external systems are now in provider packages (e.g., airflow.providers.<provider_name>.operators). The old top-level `airflow.operators` path is deprecated for these.","wrong":"from airflow.operators.s3_operator import S3Operator","symbol":"Provider specific operators (e.g., S3Operator)","correct":"from airflow.providers.amazon.operators.s3 import S3Operator"},{"note":"XComArg moved to airflow.models.xcom_arg in Airflow 3.0.","wrong":"from airflow.utils.task_group import XComArg","symbol":"XComArg","correct":"from airflow.models.xcom_arg import XComArg"}],"quickstart":{"code":"import os\nfrom datetime import datetime\n\nfrom airflow.models.dag import DAG\nfrom airflow.operators.bash import BashOperator\nfrom airflow.operators.python import PythonOperator\n\n\n# Set AIRFLOW_HOME if not already set (e.g., in a local dev setup)\n# os.environ['AIRFLOW_HOME'] = os.environ.get('AIRFLOW_HOME', '~/airflow')\n\n\ndef _greet(name):\n    print(f\"Hello, {name} from a Python task!\")\n\n\nwith DAG(\n    dag_id='simple_airflow_quickstart',\n    start_date=datetime(2023, 1, 1),\n    schedule_interval='@daily',\n    catchup=False,\n    tags=['quickstart'],\n) as dag:\n    start_task = BashOperator(\n        task_id='start_workflow',\n        bash_command='echo \"Starting the workflow!\"',\n    )\n\n    greet_task = PythonOperator(\n        task_id='greet_with_python',\n        python_callable=_greet,\n        op_kwargs={'name': 'Airflow User'},\n    )\n\n    end_task = BashOperator(\n        task_id='end_workflow',\n        bash_command='echo \"Workflow finished!\"',\n    )\n\n    start_task >> greet_task >> end_task\n","lang":"python","description":"This quickstart defines a simple DAG with Bash and Python operators. To run this locally after installing Airflow, save the code as a `.py` file (e.g., `dags/quickstart_dag.py`) in your `AIRFLOW_HOME/dags` directory. Then, initialize the database and start the Airflow standalone environment.\n\n**To set up and run Airflow locally (assuming `apache-airflow` is installed with `sqlite` support via `pip install \"apache-airflow[sqlite]\"`):**\n```bash\n# (Optional) Set AIRFLOW_HOME, e.g., to a temporary directory\nexport AIRFLOW_HOME=$(pwd)/airflow_home\n\n# Initialize the database and create an admin user (first time only)\nairflow standalone\n# Follow prompts to set admin password. This command also starts webserver, scheduler, and triggerer.\n\n# You can also start components separately:\n# airflow db migrate\n# airflow users create --username admin --firstname Airflow --lastname Admin --role Admin --email admin@example.com -p mypassword\n# airflow webserver --port 8080\n# airflow scheduler\n# airflow triggerer\n\n# After starting, visit http://localhost:8080 to enable the DAG.\n```\nRemember to define `AIRFLOW_HOME` before running `airflow standalone` or `airflow db init`."},"warnings":[{"fix":"Rewrite task code to use the Task Execution API or the Airflow Python Client for database interactions. Avoid direct SQLAlchemy imports or session usage within task logic. Consider requesting new API endpoints or Task SDK features if required functionality is missing.","message":"Direct metadata database access from task code is restricted in Airflow 3. Tasks can no longer directly import and use Airflow database sessions or models. All runtime interactions (state transitions, heartbeats, XComs, resource fetching) must now use the dedicated Task Execution API or the official Python API Client.","severity":"breaking","affected_versions":"3.0.0+"},{"fix":"Migrate existing SubDAGs to use TaskGroups for grouping related tasks, or explore using Assets and Data Aware Scheduling for more advanced scenarios.","message":"SubDAGs have been removed in Airflow 3. They are replaced by TaskGroups, Assets, and Data Aware Scheduling.","severity":"breaking","affected_versions":"3.0.0+"},{"fix":"Update Airflow configuration to use `LocalExecutor` instead of `SequentialExecutor`.","message":"The Sequential Executor has been removed in Airflow 3. It is replaced by the LocalExecutor, which can still be used with SQLite for local development.","severity":"breaking","affected_versions":"3.0.0+"},{"fix":"Remove SLA definitions from DAGs. Monitor for the introduction of 'Deadline Alerts' as a replacement.","message":"SLAs (Service Level Agreements) are deprecated and have been removed in Airflow 3. They will be replaced by forthcoming Deadline Alerts.","severity":"deprecated","affected_versions":"3.0.0+"},{"fix":"Always use full Python package paths for imports within DAGs. Ensure shared code is either installed as a Python package or added to `PYTHONPATH` with a unique top-level name to prevent clashes.","message":"Avoid using relative imports in DAG files (e.g., `from . import my_module`). The same DAG file might be parsed in different contexts (scheduler, workers, tests), leading to inconsistent behavior.","severity":"gotcha","affected_versions":"All"},{"fix":"Access Airflow Variables and Connections inside operator `execute()` methods, or pass them to operators using Jinja templating, which defers evaluation until task execution. For sensitive data, use Secrets Backend.","message":"Do not use Airflow Variables or Connections at the top level of DAG files (i.e., outside of task `execute()` methods or Jinja templates). This can cause slow DAG parsing and unexpected behavior, as the values are fetched every time the DAG file is parsed.","severity":"gotcha","affected_versions":"All"},{"fix":"Upgrade your Python environment to version 3.10 or a newer compatible version (e.g., Python 3.10, 3.11, 3.12).","message":"Apache Airflow 3.x requires Python 3.10 or newer. Attempting to install Airflow 3.x on Python 3.9 or older will result in a Python version incompatibility error during package resolution.","severity":"breaking","affected_versions":"3.0.0+"},{"fix":"Add C/C++ build tools to the Dockerfile before installing Python packages (e.g., `apk add --no-cache build-base g++` for Alpine), or use a non-Alpine Python base image (e.g., `python:3.13-slim`).","message":"Installing Apache Airflow 3.x on Alpine-based Python images (e.g., `python:3.13-alpine`) fails due to missing C/C++ build tools required by dependencies like `grpcio`. Minimal Alpine images do not include these development packages by default.","severity":"breaking","affected_versions":"3.0.0+"}],"env_vars":null,"last_verified":"2026-05-12T21:55:21.755Z","next_check":"2026-06-27T00:00:00.000Z","problems":[{"fix":"Install the missing provider package using the command: 'pip install apache-airflow-providers-microsoft-mssql'.","cause":"This error occurs when the 'apache-airflow-providers-microsoft-mssql' package is not installed, leading to missing modules required for Microsoft SQL Server operations.","error":"ModuleNotFoundError: No module named 'airflow.providers.microsoft.mssql.operators'"},{"fix":"Verify the server name, ensure the server is running, and check network connectivity. If using Airflow's Connections UI, include the full server name with domain under the host attribute.","cause":"This error indicates that the specified SQL Server is unreachable, possibly due to incorrect server details or network issues.","error":"pymssql._mssql.MSSQLDatabaseException: (20009, b'DB-Lib error message 20009, severity 9:\\nUnable to connect: Adaptive Server is unavailable or does not exist (servername)\\n')"},{"fix":"Ensure that the SQL statements are compatible with your SQL Server version and consider updating the ODBC driver to the latest version.","cause":"This error arises from SQL syntax issues, often due to compatibility problems between Airflow's ORM and the SQL Server version.","error":"sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', \"[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)\")"},{"fix":"Double-check the connection parameters, including server address, port, username, and password. Ensure the SQL Server is accessible and running.","cause":"This error suggests a failure in establishing a connection to the SQL Server, possibly due to incorrect connection parameters or server unavailability.","error":"pymssql.exceptions.OperationalError: (20002, b'DB-Lib error message 20002, severity 9:\\nAdaptive Server connection failed')"},{"fix":"Update the import statement to: 'from airflow.providers.microsoft.mssql.operators.mssql import MsSqlOperator'.","cause":"This error occurs when attempting to import 'MsSqlOperator' from an incorrect module path due to changes in Airflow's module structure.","error":"ModuleNotFoundError: No module named 'airflow.operators.mssql_operator'"}],"ecosystem":"pypi","meta_description":null,"install_score":65,"install_tag":"reviewed","quickstart_score":null,"quickstart_tag":null,"pypi_latest":"3.2.1","cli_name":"airflow","install_checks":{"last_tested":"2026-05-12","tag":"reviewed","tag_description":"minor failures on some runtimes or slightly older test data","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":0.1,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":4.87,"mem_mb":75.5,"disk_size":"229.7M"},{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":4.67,"mem_mb":75.5,"disk_size":"229.7M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":52.7,"import_time_s":4.13,"mem_mb":76.9,"disk_size":"737M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":3.56,"mem_mb":76.9,"disk_size":"737M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":24.5,"import_time_s":3.65,"mem_mb":75.5,"disk_size":"230M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":3.49,"mem_mb":75.5,"disk_size":"230M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":0.1,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":6.42,"mem_mb":81.4,"disk_size":"248.4M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":6.75,"mem_mb":81.4,"disk_size":"248.4M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":56.3,"import_time_s":6.24,"mem_mb":82.8,"disk_size":"877M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":5.46,"mem_mb":82.8,"disk_size":"877M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":23,"import_time_s":5.89,"mem_mb":81.5,"disk_size":"249M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":5.45,"mem_mb":81.4,"disk_size":"249M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":null,"import_time_s":5.81,"mem_mb":79.4,"disk_size":"238.7M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":5.79,"mem_mb":79.3,"disk_size":"238.7M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":44.7,"import_time_s":6.28,"mem_mb":80.8,"disk_size":"868M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":6.22,"mem_mb":80.8,"disk_size":"868M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":"wheel","failure_reason":null,"install_time_s":18.3,"import_time_s":6.01,"mem_mb":79.4,"disk_size":"240M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":6.05,"mem_mb":79.4,"disk_size":"240M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":13.8,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":10.7,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":2.1,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"celery,cncf.kubernetes,http,postgres,amazon","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":"build_error","install_time_s":2.3,"import_time_s":null,"mem_mb":null,"disk_size":null},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":1,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":null,"mem_mb":null,"disk_size":null}]},"quickstart_checks":{"last_tested":"2026-04-24","tag":null,"tag_description":null,"results":[{"runtime":"python:3.10-alpine","exit_code":-1},{"runtime":"python:3.10-slim","exit_code":-1},{"runtime":"python:3.11-alpine","exit_code":-1},{"runtime":"python:3.11-slim","exit_code":-1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":-1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}