Apache Airflow Alibaba Cloud Provider

3.3.7 · active · verified Sun Apr 12

The Apache Airflow Alibaba Cloud Provider enables integration with various Alibaba Cloud services such as Object Storage Service (OSS), DataWorks, DirectMail, and more. It provides hooks, operators, and sensors to incorporate Alibaba Cloud workflows into Airflow DAGs. This provider is currently at version 3.3.7 and receives updates typically aligned with Apache Airflow's release cycle or as individual feature/bugfix releases.

Warnings

breaking Version 3.0.0 introduced significant internal changes, including the relocation of `AlibabaCloudBaseHook` from `airflow.providers.alibaba.cloud.connection` to `airflow.providers.alibaba.cloud.hooks.base`. This impacts custom hooks or operators inheriting directly from the base hook.
Fix: Review custom integrations inheriting from `AlibabaCloudBaseHook` and adapt to the new import path and connection architecture. Direct instantiation of `AlibabaCloudBaseHook` is generally not recommended.
gotcha Alibaba Cloud connections require specific configuration fields in Airflow. The `Conn Type` must be 'Alibaba Cloud', and `Login` (Access Key ID), `Password` (Access Key Secret), and `Extra` (`{"region": "your-region"}`) fields are critical. Incorrect region configuration is a common cause of 'service not available' or authentication errors.
Fix: Ensure the Airflow connection (e.g., `alibaba_cloud_default`) is correctly configured, including the `Conn Type`, credentials, and the exact region (`cn-hangzhou`, `us-east-1`, etc.) where your Alibaba Cloud resources are provisioned.
gotcha The Alibaba Cloud RAM user associated with the Access Key ID and Secret must possess the necessary IAM permissions for the specific Alibaba Cloud service being accessed (e.g., `DirectMail:SendMail`, `OSS:PutObject`). Missing permissions will result in 'Access Denied' errors from the service.
Fix: Verify that the RAM policy attached to the user or role used for authentication grants the required actions on the target Alibaba Cloud services and resources.
gotcha Many Alibaba Cloud services are regional. Ensure that the `region` parameter supplied to operators or defined in the Airflow connection matches the region where your specific Alibaba Cloud resource (e.g., OSS bucket, DirectMail sender domain, DataWorks workspace) exists. Cross-region access for some services may be restricted or require specific configurations.
Fix: Explicitly specify the correct region in the Airflow connection's 'Extra' field or directly in the operator's arguments to match the location of your Alibaba Cloud resources.

Install

pip install apache-airflow-providers-alibaba Install latest version

Imports

DirectMailSendMailOperator

from airflow.providers.alibaba.cloud.operators.directmail import DirectMailSendMailOperator

OSSCopyObjectOperator

from airflow.providers.alibaba.cloud.operators.oss import OSSCopyObjectOperator

AlibabaCloudBaseHook
```
from airflow.providers.alibaba.cloud.hooks.base import AlibabaCloudBaseHook
```
The base hook was moved to `hooks.base` in versions >=3.0.0; direct use is discouraged for internal reasons.

Quickstart

This quickstart demonstrates sending an email using the `DirectMailSendMailOperator`. It requires an 'Alibaba Cloud' connection named `alibaba_cloud_default` to be configured in Airflow with appropriate credentials and region. Replace placeholder emails and ensure the sender is a verified DirectMail account.

from __future__ import annotations

import pendulum

from airflow.models.dag import DAG
from airflow.providers.alibaba.cloud.operators.directmail import DirectMailSendMailOperator

# Before running this DAG:
# 1. Create an Airflow Connection:
#    - Conn Id: 'alibaba_cloud_default'
#    - Conn Type: 'Alibaba Cloud'
#    - Host: 'https://dm.cn-hangzhou.aliyuncs.com' (or your DirectMail endpoint)
#    - Login: YOUR_ALIYUN_ACCESS_KEY_ID
#    - Password: YOUR_ALIYUN_ACCESS_KEY_SECRET
#    - Extra: {'region': 'cn-hangzhou'} (or your service region)
# 2. Ensure 'account_name' is a verified sender email in your Alibaba DirectMail console.
# 3. Replace 'recipient@example.com' with an actual recipient.

with DAG(
    dag_id="alibaba_directmail_send_example",
    schedule=None,
    start_date=pendulum.datetime(2023, 10, 26, tz="UTC"),
    catchup=False,
    tags=["alibaba", "directmail", "example"],
) as dag:
    send_mail_task = DirectMailSendMailOperator(
        task_id="send_test_email",
        conn_id="alibaba_cloud_default",
        account_name="no-reply@your-domain.com",  # Your verified sender address
        from_alias="Airflow Notification",
        to_address="recipient@example.com",      # Recipient email
        subject="Airflow Test Email from Alibaba DirectMail",
        html_body="This is a test email sent from Apache Airflow using Alibaba Cloud DirectMail.",
        region="cn-hangzhou",                    # Must match your DirectMail service region
    )

view raw JSON →