Datadog Provider for Apache Airflow

raw JSON →
3.10.3 verified Sat Apr 25 auth: no python

This provider package integrates Apache Airflow with Datadog, enabling users to send metrics, events, and query metrics from Datadog directly within their Airflow DAGs. It supports monitoring of DAGs, tasks, and other Airflow components through Datadog's platform. The current version is 3.10.3 and it follows the Apache Airflow provider release cadence, often aligning with Airflow core releases but also releasing independently for bug fixes and features.

pip install apache-airflow-providers-datadog
error ModuleNotFoundError: No module named 'airflow.providers.datadog'
cause The 'apache-airflow-providers-datadog' package is not installed.
fix
Install the package using 'pip install apache-airflow-providers-datadog'.
error ImportError: cannot import name 'DatadogHook' from 'airflow.providers.datadog.hooks.datadog'
cause The import statement is incorrect due to changes in module structure.
fix
Use 'from airflow.providers.datadog.hooks.datadog import DatadogHook' instead.
error AttributeError: module 'airflow.providers.datadog.hooks.datadog' has no attribute 'DatadogHook'
cause The 'DatadogHook' class has been moved or renamed in the module.
fix
Ensure you are using the correct import path: 'from airflow.providers.datadog.hooks.datadog import DatadogHook'.
error Broken DAG: [/path/to/dag.py] No module named 'datadog'
cause The 'datadog' Python package is not installed, which is a dependency for the Datadog provider.
fix
Install the 'datadog' package using 'pip install datadog'.
error TypeError: DatadogHook() missing 1 required positional argument: 'datadog_conn_id'
cause The 'datadog_conn_id' parameter is required when initializing 'DatadogHook'.
fix
Initialize 'DatadogHook' with the required 'datadog_conn_id' parameter: 'DatadogHook(datadog_conn_id="your_connection_id")'.
breaking Provider version 3.10.0+ requires Apache Airflow 2.11.0+. Older provider versions have different minimum Airflow requirements (e.g., 3.0.0 requires Airflow 2.2+, 2.0.0 requires Airflow 2.1.0+). Always check the provider's changelog for specific Airflow compatibility when upgrading.
fix Upgrade your Apache Airflow instance to at least the minimum version required by the provider, or pin the provider version to one compatible with your Airflow installation.
breaking The import paths for all provider classes (Hooks, Operators, Sensors) changed from `airflow.contrib.*` or legacy provider paths to `airflow.providers.datadog.*` with the introduction of Airflow 2.0 and the unified provider package system.
fix Update import statements in your DAGs to use the new `airflow.providers.datadog` namespace (e.g., `from airflow.providers.datadog.hooks.datadog import DatadogHook`).
gotcha For comprehensive monitoring (metrics, logs, traces), simply installing the provider is not enough. You also need to configure the Datadog Agent, enable Airflow's StatsD plugin in `airflow.cfg`, and potentially set up log collection and OpenLineage.
fix Refer to the official Datadog documentation for a complete Airflow integration setup, including Agent installation, `airflow.cfg` modifications for StatsD, and log/trace collection configurations.
gotcha Using too many unique tags or high-cardinality tags (e.g., dynamic, unbounded values) when sending metrics to Datadog can significantly increase your Datadog bill and impact performance.
fix Design your tags carefully, keeping them concise and using a finite set of values. Avoid using highly dynamic or unique values (like `UUID`s or full timestamps) as tags. Aggregate metrics where appropriate to reduce cardinality.
gotcha When using `DatadogOperator` or `DatadogHook`, ensure your Datadog API and APP keys are securely configured in Airflow Connections (recommended) or via environment variables accessible to the Airflow worker. Hardcoding credentials in DAG files is a security risk.
fix Create an Airflow connection of type 'Datadog' (or 'HTTP' for older versions) with `conn_id='datadog_default'` (or a custom ID) and store your API/APP keys there. Reference this `conn_id` in your operators/hooks.
runtime status import time mem disk
3.10-alpine 4.82s 68.8MB 245.1M
3.10-slim 3.55s 68.8MB 243M
3.11-alpine 7.03s 74.5MB 265.2M
3.11-slim 5.49s 74.6MB 263M
3.12-alpine 6.08s 73.3MB 255.2M
3.12-slim 6.82s 73.3MB 254M
3.13-alpine 5.73s 73.9MB 257.0M
3.13-slim 7.07s 73.9MB 256M
3.9-alpine 5.20s 60.7MB 211.9M
3.9-slim 5.94s 60.7MB 207M

This example demonstrates how to use the `DatadogOperator` to send a custom metric to Datadog. It assumes that Datadog API and APP keys are configured either via an Airflow connection named `datadog_default` or directly through environment variables `DATADOG_API_KEY` and `DATADOG_APP_KEY`.

import os
from datetime import datetime

from airflow import DAG
from airflow.providers.datadog.operators.datadog import DatadogOperator

# Datadog API and APP keys are typically configured as an Airflow Connection
# with conn_id='datadog_default'. For demonstration, using environment vars.
# In a real Airflow environment, prefer Airflow Connections for credentials.

with DAG(
    dag_id='datadog_metrics_example',
    start_date=datetime(2023, 1, 1),
    schedule_interval=None,
    catchup=False,
    tags=['datadog', 'metrics'],
    params={
        'metric_value': 123.45
    }
) as dag:
    send_custom_metric_task = DatadogOperator(
        task_id='send_custom_metric',
        datadog_conn_id='datadog_default', # Ensure this connection is configured in Airflow UI
        metric_name='my.airflow.custom.metric',
        metric_type='gauge',
        points=[["{{ execution_date.int_timestamp }}", "{{ params.metric_value }}"]],
        tags=['env:dev', 'dag_id:{{ dag.dag_id }}', 'task_id:{{ ti.task_id }}'],
        # API/APP keys can be passed here, but Airflow Connections are preferred
        api_key=os.environ.get('DATADOG_API_KEY', ''),
        app_key=os.environ.get('DATADOG_APP_KEY', ''),
    )

    # Note: For this DAG to run successfully and send metrics to Datadog,
    # you must have a Datadog connection configured in Airflow UI (Admin -> Connections)
    # with Conn Id 'datadog_default' and provide your Datadog API and APP keys.
    # Alternatively, ensure DATADOG_API_KEY and DATADOG_APP_KEY environment
    # variables are set where the Airflow worker executes the task.