Dagster Datadog Integration

0.29.0 · active · verified Thu Apr 16

The `dagster-datadog` library provides components to integrate Dagster with Datadog for monitoring and observability. This includes resources for sending metrics and events, and loggers for forwarding Dagster logs to Datadog. It is currently at version 0.29.0 and is released as part of the Dagster ecosystem's approximately monthly release cycle, with library versions tracking core Dagster releases.

Common errors

```
ModuleNotFoundError: No module named 'dagster_datadog'
```
cause The `dagster-datadog` library has not been installed in your Python environment.

fix Run `pip install dagster-datadog`.
```
dagster._core.errors.DagsterInvalidDefinitionError: Resource with key 'datadog' expected but not provided.
```
cause Your job or asset tries to access a resource named 'datadog', but it hasn't been defined in your `Definitions` or provided to the job/graph.

fix Add `DatadogResource` to your `Definitions` resources, e.g., `resources={'datadog': DatadogResource(...)}}`.
```
dagster._core.errors.DagsterInvalidConfigError: Missing required config entry 'api_key'.
```
cause The `DatadogResource` or `DatadogLogger` was initialized without providing the necessary `api_key` configuration.

fix Provide `api_key` in the resource or logger configuration, e.g., `DatadogResource(api_key="YOUR_API_KEY", app_key="YOUR_APP_KEY")`.
```
AttributeError: module 'dagster_datadog' has no attribute 'datadog_resource'
```
cause You are trying to import or use the old function-based `datadog_resource` which was removed or deprecated in Dagster 1.0+.

fix Update your code to use the class-based `DatadogResource` instead: `from dagster_datadog import DatadogResource`.

Warnings

breaking Dagster 1.0+ introduced a new resource API. The function-based `datadog_resource` is now deprecated and should be replaced with the class-based `DatadogResource`.
Fix: Replace `datadog_resource()` with `DatadogResource()` when defining your resources, and provide configuration parameters directly to the class constructor.
gotcha The `DatadogResource` and `DatadogLogger` require Datadog `api_key` and `app_key` for successful authentication and data submission. Misconfiguration or omission of these keys will lead to failed Datadog interactions.
Fix: Ensure `api_key` and `app_key` are provided to `DatadogResource` and `DatadogLogger` constructors, preferably using environment variables (e.g., `os.environ.get("DATADOG_API_KEY")`).
gotcha When sending metrics directly to the Datadog Agent (default behavior if `host`/`port` point to an agent), ensure the agent is running and accessible from the Dagster execution environment. Otherwise, configure the resource to use the HTTP API directly.
Fix: Verify Datadog Agent status or explicitly configure `host` and `port` in `DatadogResource` if using a non-default agent location, or ensure `api_key`/`app_key` are provided for direct HTTP API usage.

Install

pip install dagster-datadog Install dagster-datadog

Imports

DatadogResource
wrong:
```
from dagster_datadog import datadog_resource
```
correct:
```
from dagster_datadog import DatadogResource
```
The function-based `datadog_resource` was deprecated in Dagster 1.0+ in favor of the class-based `DatadogResource`.
DatadogLogger
```
from dagster_datadog import DatadogLogger
```
For sending Dagster system logs to Datadog.

Quickstart

This quickstart demonstrates defining a Dagster asset that interacts with `DatadogResource` to send a custom metric. It shows how to configure the `DatadogResource` and `DatadogLogger` within your Dagster definitions, requiring Datadog API and APP keys typically provided via environment variables.

from dagster import AssetExecutionContext, Definitions, asset, job
from dagster_datadog import DatadogResource, DatadogLogger
import os

@asset
def my_datadog_asset(context: AssetExecutionContext, datadog: DatadogResource):
    context.log.info("Asset started, sending metric to Datadog...")
    # Use the DatadogResource client to send a custom metric
    datadog.send_metric(
        metric_name="my_asset.completion",
        value=1,
        tags=["asset:my_datadog_asset", f"run_id:{context.run_id}"],
    )
    context.log.info("Metric sent. Asset completed.")

@job
def my_datadog_job():
    my_datadog_asset()

defs = Definitions(
    assets=[my_datadog_asset],
    resources={
        "datadog": DatadogResource(
            api_key=os.environ.get("DATADOG_API_KEY", ""),
            app_key=os.environ.get("DATADOG_APP_KEY", ""),
            # Optional: configure host/port if Datadog Agent is not on localhost
            # host="localhost",
            # port=8125,
        )
    },
    # Optional: configure a Datadog logger for system events
    loggers={
        "datadog": DatadogLogger(
            api_key=os.environ.get("DATADOG_API_KEY", ""),
            app_key=os.environ.get("DATADOG_APP_KEY", "")
        )
    }
)

# To run this example:
# 1. Ensure DATADOG_API_KEY and DATADOG_APP_KEY environment variables are set.
# 2. Save as `repo.py` and run `dagster dev -f repo.py`.
# 3. Launch `my_datadog_job` from the UI.

view raw JSON →