Datadog Provider for Apache Airflow

3.10.3 · active · verified Sat Apr 11

This provider package integrates Apache Airflow with Datadog, enabling users to send metrics, events, and query metrics from Datadog directly within their Airflow DAGs. It supports monitoring of DAGs, tasks, and other Airflow components through Datadog's platform. The current version is 3.10.3 and it follows the Apache Airflow provider release cadence, often aligning with Airflow core releases but also releasing independently for bug fixes and features.

Warnings

breaking Provider version 3.10.0+ requires Apache Airflow 2.11.0+. Older provider versions have different minimum Airflow requirements (e.g., 3.0.0 requires Airflow 2.2+, 2.0.0 requires Airflow 2.1.0+). Always check the provider's changelog for specific Airflow compatibility when upgrading.
Fix: Upgrade your Apache Airflow instance to at least the minimum version required by the provider, or pin the provider version to one compatible with your Airflow installation.
breaking The import paths for all provider classes (Hooks, Operators, Sensors) changed from `airflow.contrib.*` or legacy provider paths to `airflow.providers.datadog.*` with the introduction of Airflow 2.0 and the unified provider package system.
Fix: Update import statements in your DAGs to use the new `airflow.providers.datadog` namespace (e.g., `from airflow.providers.datadog.hooks.datadog import DatadogHook`).
gotcha For comprehensive monitoring (metrics, logs, traces), simply installing the provider is not enough. You also need to configure the Datadog Agent, enable Airflow's StatsD plugin in `airflow.cfg`, and potentially set up log collection and OpenLineage.
Fix: Refer to the official Datadog documentation for a complete Airflow integration setup, including Agent installation, `airflow.cfg` modifications for StatsD, and log/trace collection configurations.
gotcha Using too many unique tags or high-cardinality tags (e.g., dynamic, unbounded values) when sending metrics to Datadog can significantly increase your Datadog bill and impact performance.
Fix: Design your tags carefully, keeping them concise and using a finite set of values. Avoid using highly dynamic or unique values (like `UUID`s or full timestamps) as tags. Aggregate metrics where appropriate to reduce cardinality.
gotcha When using `DatadogOperator` or `DatadogHook`, ensure your Datadog API and APP keys are securely configured in Airflow Connections (recommended) or via environment variables accessible to the Airflow worker. Hardcoding credentials in DAG files is a security risk.
Fix: Create an Airflow connection of type 'Datadog' (or 'HTTP' for older versions) with `conn_id='datadog_default'` (or a custom ID) and store your API/APP keys there. Reference this `conn_id` in your operators/hooks.

Install

pip install apache-airflow-providers-datadog Install latest version

Imports

DatadogHook
```
from airflow.providers.datadog.hooks.datadog import DatadogHook
```
Old import path for Airflow 1.x and early 2.x 'backport' providers; current providers use `airflow.providers` namespace.
DatadogOperator
```
from airflow.providers.datadog.operators.datadog import DatadogOperator
```
The `DatadogMonitorOperator` was seen in older examples (pre-2021); the current convention uses `DatadogOperator` directly under the provider's `operators` module.
DatadogSensor
```
from airflow.providers.datadog.sensors.datadog import DatadogSensor
```
Old import path for Airflow 1.x and early 2.x 'backport' providers; current providers use `airflow.providers` namespace.

Quickstart

This example demonstrates how to use the `DatadogOperator` to send a custom metric to Datadog. It assumes that Datadog API and APP keys are configured either via an Airflow connection named `datadog_default` or directly through environment variables `DATADOG_API_KEY` and `DATADOG_APP_KEY`.

import os
from datetime import datetime

from airflow import DAG
from airflow.providers.datadog.operators.datadog import DatadogOperator

# Datadog API and APP keys are typically configured as an Airflow Connection
# with conn_id='datadog_default'. For demonstration, using environment vars.
# In a real Airflow environment, prefer Airflow Connections for credentials.

with DAG(
    dag_id='datadog_metrics_example',
    start_date=datetime(2023, 1, 1),
    schedule_interval=None,
    catchup=False,
    tags=['datadog', 'metrics'],
    params={
        'metric_value': 123.45
    }
) as dag:
    send_custom_metric_task = DatadogOperator(
        task_id='send_custom_metric',
        datadog_conn_id='datadog_default', # Ensure this connection is configured in Airflow UI
        metric_name='my.airflow.custom.metric',
        metric_type='gauge',
        points=[["{{ execution_date.int_timestamp }}", "{{ params.metric_value }}"]],
        tags=['env:dev', 'dag_id:{{ dag.dag_id }}', 'task_id:{{ ti.task_id }}'],
        # API/APP keys can be passed here, but Airflow Connections are preferred
        api_key=os.environ.get('DATADOG_API_KEY', ''),
        app_key=os.environ.get('DATADOG_APP_KEY', ''),
    )

    # Note: For this DAG to run successfully and send metrics to Datadog,
    # you must have a Datadog connection configured in Airflow UI (Admin -> Connections)
    # with Conn Id 'datadog_default' and provide your Datadog API and APP keys.
    # Alternatively, ensure DATADOG_API_KEY and DATADOG_APP_KEY environment
    # variables are set where the Airflow worker executes the task.

view raw JSON →